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Abstract 


As the premier leader in space technology research and development, NASA is investing in new 
technologies (1) that include 14 primary technology roadmap areas, and aeronautics (2) . Understanding the 
cost for research and development of these technologies and the time it takes to increase the maturity of 
the technology is important to the support of the ongoing and future NASA missions. Specifically, there 
exists a need to provide estimating capabilities for technologies in the Technology Readiness Level 
(TRL) range of 2 to 6. Overall, technology estimating may help provide guidance to technology 
investment strategies initially for NASA’s Game Changing Development Program Office and similar 
stakeholders help improve evaluation of technology affordability, and aid in decision support. 

A year-long research effort was initiated to help identify a systematic process for estimating the 
cost and schedule of low TRL technology research and development projects. This research project effort 
was complimented by phased work group sessions, multiple meetings, data acquisition and analysis, and 
discussions with experts in the field of technology maturation, NASA executives, technologists, analyst, 
principal investigators, engineers, and project/program managers. 

This research report provides a summary of the framework development of a Technology 
Estimating process where four technology roadmap areas were initially selected to be studied. The 
framework includes definition of terms, discussion for narrowing the focus from 14 NASA Technology 
Roadmap areas to four, and further refinement to include technologies in the TRL range of 2 to 6. 
Included in this paper is a discussion to address the evaluation of 20 unique technology parameters that 
were initially identified, evaluated and then subsequently reduced in number determined suitable for use 
in characterizing these technologies. Further, a discussion of data acquisition effort and criteria 
established for data quality are provided. A summary of the findings obtained during the research 
including gaps identified, description of a spreadsheet-based estimating tool initiated as a part of the 
Technology Estimating process, and the examination of a case study are provided. Suggestions for 
continued investigations in this research are also addressed. 
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Introduction and Background 

The need for technology estimating was addressed to the NASA Cost community in 
201 ICost Symposium, where Sefcik et ah, 2011 identified: 

• No current Agency/Glenn Research Center database exists for estimating technology 
development projects, 

• Data to populate such a database is in technologist’s head or spread out amongst many 
people or in long term storage or missing/discarded due to personnel turnover or . . 

• No known good method to estimate the cost of Technology Reference Level (TRL)* 
advancement that is supported by actual data, 

• Reduction in Agency technology projects since 2005 results in little recent data that 
would be available, and 

• Estimates tend to be worked to the year’s budget versus planning through project 
completion. (3) 

Note: * = TRL is defined in Appendix A. 

In response to the NASA Cost Community’s call for implementation of a capability to provide 
technology estimating, this research project was initiated and sponsored by the Office of Evaluation, Cost 
Analysis Division in collaboration with the Space Technology Mission Directorate, Game Changing 
Development Program Office. The goal was to improve NASA’s ability to estimate the cost of 
technology development and maturation. This technology estimating research involves identifying 
parameters that are expected to influence the cost of a technology, collecting actual data for completed 
technology efforts, and then conducting mathematical and statistical analyses to generate cost estimating 
relationships between the parameters and use these in an analysis to determine the cost and schedule of 
technology development. (4) 

The procedural process diagram for the conduct of the research was prepared by the Cost 
Analysis Division and presented in Figure 1. The process was identified to include a series of workshops 
and internal reviews within the NASA Cost Community to elucidate issues and the development of the 
Technology Estimating research, provide the necessary community input and acceptance of the process, 
and provide a platform for the required internal peer review. 
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Figure 1. Technology Estimating Process Diagram, 2012-2013 


Purpose 

Technology has been defined in a number of ways. NASA defines it as: 

“a solution that arises from applying the disciplines of engineering science to synthesize a device, 
process, or subsystem, to enable a specific capability .” 

Industry definitions include: 

By Boer, 1999 “the application of knowledge to useful objectives. It is usually built on previous 
technology by adding new technology inputs or new scientific knowledge ” (6) , or by Van Wyk, 1999 
“Created competence as manifested in devices, procedures, and acquired human skills ”. (7) 

Fundamentally, given any definition, it is recognized that technology will possess a monetary cost 
for its research and development, and will exhibit the duration in time to achieve its full maturation in 
preparation for inclusion in subsequent flight projects. Both are fundamental metrics that should be 
known. What is the purpose then for Technology Estimating, what are the benefits, and how do we do it? 
The intent of this paper is to report the information accumulated and process developed so as to document 
the technology estimating effort. 

Beltramo, 1988 identified the primary purpose of a cost estimating model is to support a decision 
at some level. The choice may be a “go” or “no go”, alternative A versus B, design trade off at the 
component level, or at some other level. Under all circumstances, the user must carefully consider the 
assumptions inherent in the estimating to determine they are appropriate for the intended purpose. 1 (8) 
Gaining an understanding of the cost and schedule of developing a technology could be critical for those 
who acquire technology research and development or to those who need to manage specific technology 
projects and technology advancement programs that manage a portfolio of technology projects. Clearly, 
an increased understanding of technology cost(s) or when management is involved in technology 
investments and how money may be spent on a single project, or either in a limited or broad portfolio of 
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specific technologies, technology estimating should help in decision making. Oberlender et ah, 2001, 
recognized that early estimating of projects is vital for business unit decisions that include strategies for 
asset development, potential project screening, and resource commitments. (9) 

Benefits 

We must acknowledge that the motivation for delivering or pursuing new technology research 
and development activities may be distinctly different for each stakeholder, and each has an opportunity 
to enjoy different benefits. NASA benefits were identified and discussed with technologists and the cost 
estimating community for this research and they are presented in Table 1. 


Table 1 

Benefits to NASA - Technology Estimating 


NASA Benefit 


Pre-Phase A, to satisfy 
NPR7120.5E, 5.f. Basis 
of Estimate (cost and 
schedule) 

“The project also develops and maintains the status of a set of programmatic and technical 
leading indicators to ensure proper progress and management of the project. These 
include cost trends”. 

Analyze technology 
costs to satisfy NPR 
g 7120.5E, 8.0 
g Technology Readiness 
kQ Assessment and 
O Development 

“Provide technology development schedules, including intermediate milestones and 
funding requirements, during Phases A and B for each identified technology development 
to achieve TRL 6 by PDR. Describe expected status of each technology development at 
SRR, MDR/SDR, and PDR.” 

^ Analyze technology 
§ costs to satisfy NPR 
m 7120.5E, 3.5 
g Technology 
* q> Development Plan 
where it is required 

“describe how the project will transition technologies from the development stage to the 
manufacturing and production phases.” Technology Estimating would provide the data to 
evaluate the cost and schedule during the phases. 

Support NPR 7120.8 
NASA Research and 
Technology Program 
and Project 
Management 
Requirements, R&T 
program Commitment 
Agreement (PCA) 
Template 

Provide the estimated cost range for the program for the five-year period beginning in the 
current fiscal year at a level of detail that identifies the approved individual projects. 
Identify the constraints and assumptions used to develop this estimated cost range and 
specifically identify those assumptions that drive the range. This cost range should contain 
all costs necessary to perform the program, including, but not limited to, standard project 
activities, required technology developments, facilities costs, infrastructure costs, 
operations and sustainment, data analysis, and disposal. Reference the annual budget 
contained in the Integrated Budget and Performance Document (IBPD) for cost phasing. 
The cost range should be updated when program content changes, such as the addition of 
new projects entering implementation.” Appendix D, 6.0 Cost Commitment 

Task Managers, Program 
Managers, Principal 
Investigators 

Provide capability and dataset to benchmark cost and schedule estimate plans against prior 
technology investments. 

Increase confidence in technology cost/schedule thus increasing capability to manage 
technology costs. 

Help analyze factors that influence technology cost and thereby reduce cost. 

Technologists 

Provide a capability and dataset to better anticipate the cost and schedule resources needed 
to mature a technology to a target TRL level. 

NASA-wide 

Supports need to independently assess the likelihood that various technology cost and 
schedule plans can be successfully met. 

Provides a tool and dataset that addresses gap in existing cost analysis methods for 
technologies TRL 2 to 6. 

Knowledge of Technology cost(s) and schedule increases understanding as to whether it 
can be affordable. 

Establishes framework for future data collection to further enhance estimating capabilities. 
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How you perform technology cost and schedule estimating can be a challenge. The initial 
challenge comes from the decision for which cost model or estimating process should be selected. The 
considerations made for selection must include what ultimate process is best for evaluating complex 
technologies, each with their own set of unique or potentially abstract conditions. Added to the challenge 
are potential differences in the characterization of a technology, or from one technology to another where 
the difference can be significant. 

The difficulties and considerations made, for how to estimate technology has been addressed by 
Hazelrigg et al., 1991 who recognized the difficulties in estimating the cost of space technologies (10) , Roy 
et al., 2005 recognized the need in commercial applications where a systematic approach to cost 
estimating of technologies in the automotive industry was needed (11) , Ingalls et al., 2004 who recognized 
that the use of parametric estimating in budgeting, scheduling, and control of (Technology driven) 
projects would enhance the ability of project management organizations to effectively and efficiently 
utilize valuable resources, (12) and Cyr, 1988 who investigated parametric cost estimating methods for 
advanced space systems in the conceptual phase. (13) The Rand Corporation, 1992 while engaged in 
technology estimating of complex aerospace technologies, identified the need to perform estimating using 
parametric analysis for complex systems early in the development of advanced technologies. (14) From this 
community, it was evident that a technology, during its early stages of research and development, must be 
characterized, not only by its maturity, but by how much definition a technology was given by a thorough 
characterization. A basic assumption taken in this research was that all technologies have inherent 
variability . 

Determination of a Cost Estimating Process 

Knowing that there was variability in the very thing we are trying to estimate, and given 14 plus 
one technology areas, the derivation of credible estimates seems to be a daunting task. How then do we 
begin to select a cost estimating process and what NASA technologies do we address? To answer these 
questions for technology estimating in this research, the Research Team proceeded in two steps: 1) 
summarize the types of cost estimating processes available; then 2) to make simple refinement of the 
NASA technology data based on TRL. In the first step, a short summary has been identified for five cost 
estimating processes, and they are presented in Table 2. 


Table 2 

Cost Estimating Processes - Advantages and Disadvantages 
(Hazelrigg et al. 1991 <10) , Roy Et al. 2005 (11) , Shishko et al. 2003 <15) , Dumas 1979 <16) , Cavalieria, 2003 <17) ) 

Cost Estimating Process 

Advantages 

Disadvantages 

Analogy-based technique (10) 

Estimate is based on degree of similarity between 
projects. 

Used when little or no data is available. 

Degree of similarity is difficult to assess. Costs 
are subjectively adjusted up or down depending 
on the project complexity. 

Engineering approach (11) 
(also known as “bottoms up”) 

Method is systematic and based on elementary 
components. 

Design or technology must be very well defined. 
Requires large amount of time for preparation. 

Real Options Valuation (15) 

Valuing projects in an environment of extreme 
uncertainty. 

Requires extensive systems thinking and 
engineering analysis to build a credible 
simulation of alternative sample paths. 

Parametric (16) 

Objective, unbiased, consistent. 

Takes less time. 

Suitable for use early in project formulation with 
limited project definition and limited data 

Assumption is that same condition(s) which 
governed costs in the past will be the same on 
each project. 

May exacerbate cost growth. 

Artificial neural network (17) 

Non-linear and parallel, can “infer” from knowledge. 

Requires training, and provides “black box” 
solutions 
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After a literature review and review of the multiple analytical methods, the formalized parametric 
estimating techniques and data collection and analysis as outlined by the International Cost Estimating 
and Analysis Association in the Parametric Estimating Handbook (18) was determined to be the best option 
for further investigation. The selection of parametric estimating as the best choice for use in cost 
estimating technology is also substantiated by Thibault, 1992 who indicated that the “US Defense 
Contract Audit Agency believes now, as it always believed, that parametric estimating techniques using 
cost-estimating relationships are acceptable in the appropriate circumstances for proposing costs on 
government contracts. ” (19) Fad et al., 1988, identified that parametrics is only the cost estimating method 
that can function within the limited data and estimating turn-around time constraints of the new business 
environment. (20) The Federal Acquisition Regulations (FAR) 1 5.404-1 (b)(2)(iii) and 4(c)(i)(C) states that 
parametrics are an acceptable method for performing price analysis. (21) Based on the comparison of 
advantages and disadvantages of various accepted estimating processes and literature review, parametric 
estimating provides the best possible solution for satisfying this technology estimating research given 
the need to perform analysis early in project definition and possessing limited data. 

Scope of Technology 

The second step in the cost estimating process is that we refined the NASA technologies by 
separating them into two classes, i.e. TRL 2 to 6; and technologies greater than TRL 6. This separation of 
technologies into classes was made to simplify and reduce the scope of technologies to be investigated. It 
is commonly recognized that technologies in the TRL range of 2 to 6 normally have limited detailed 
project information, while the extent of detail or information for technologies with TRL > 6 usually 
possesses project information in much greater detail. Technologies with TRL > 6 were also identified to 
have significant or extensive amount of project related information and this information has been 
typically and accumulated in and represented by many cost estimating cost models and data sets adopted 
by NASA Cost Community such as NAFCOM and REDSTAR. The investigation of TRL 1 projects was 
deferred in this initial study to help limit the number of potential projects and reduce the scope of the 
research effort. 

In the Phase I Workshop, technologies with TRL range from 2 to 6 were identified to include 
those systems, sub-systems, parts, components and materials that were non- flight technologies. Figure 2 
presents the primary scope of technologies that were investigated and provides a matrix indicating the 
technologies of interest in this research study were TRL in the range of 2 to 6. 
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Figure 2. TRL Versus Technology Scope Classification - Presented at the Phase I Workshop 


Before the Phase I Workshop, the Research Team deliberated on the extent of data that may be 
available in each of the respective technology areas and that several challenges may exist. Later these 
challenges were found to become evident during the development and investigation of the TRL 2 to 6 
technology projects scope. Each of these “challenges” is listed below with a Research Team observation: 


1. Technology Tracking Not Standardized 

Meaning: Standardization of Technology Tracking at NASA would allow an understanding of 

technology and TRL progression and help refine an understanding of native problems or difficulties in 
certain technologies, and to help identify common issues. Tracking of technologies in successive phases 
or similar technologies between programs would help to understand the progression of a technology, and 
possible impacts to duration. 

Research 

Observations: 

a. The deployment of NASA’s technology roadmap was not brought to light until at least the period of 
time after February 2012, when the Steering Committee for NASA Technology Roadmaps; National 
Research Council of the National Academies published “ NASA Space Technology Roadmaps and 
Priorities : Restoring NASA's Technological Edge and Paving the Way for a New Era in Space ”. (2) 
After this publication, NASA’s technologies would then be assigned into specific Technology Areas 
(TAs) and these assignments were in the early process of being established. Prior to this publication 
therefore, no previous technology project data before February 2012 would be expected to 
consistently contain the formalized NASA Technology Area Roadmap designations. 
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b. NASA’s new technology information resource ( TechPort ) was published internally to NASA on 
October 10, 2012. TechPort was constructed for tracking technologies and providing help 
standardization of the tracking of technology. After an initial survey of the data contained in TechPort 
during the first phase of this research project, only five completed projects were found to be included 
in the first publication, although this number was found to grow with time. Further, only five of the 
20 technology estimating parameters were found common in TechPort, and many of these were not 
fully completed and provided as placeholder to be completed later. 

c. Tracking technology in its early progressive stages along the path of development or where the 
early technology has branched to other Technology Roadmap Areas (TAs) would be a significant 
building block to Technology Estimating. Systematic tracking of technology cost and schedule 
efforts using the technology characterization parameters identified in this research as they could be 
applied to NASA and other data sources. These data sources may include the New Technology 
Reporting https://invention.nasa.gov/, TechPort https://techport.nasa.gov/techport/home.action . Small 
Business Innovation Research/ Small Business Technology Transfer (SBIR/STTR) 
http://sbir.gsfc.nasa.gov/SBIR/SBIR.html , Earth Science Technology Office (ESTO) Techfolio 
http://www.estotechnology.us/techportfolio/ . One NASA Cost Engineering (ONCE), and other 
governmental organizations. 

2. Naming system or Taxonomy of Technologies 

Meaning: A naming system for technologies helps to identify where a technology fits or is 

attributed to the larger system or mission. Whether the particular technology is a material, part, 
component, etc. of a larger system, a naming system or convention was perceived to be especially helpful 
when multiple items are included in the construct of the technology. 

Research 

Observations: 

a. The NASA Systems Engineering Handbook identifies systems, components, etc. as shown ballooned 
in red in Figure 3. (5) Beyond the Work Breakdown Structure (WBS), which is unique to each 
technology, NASA does not appear to have a standard taxonomy to name overall systems or 
components. Further, with respect this research, technology projects in the range of TRL 2 to 6 do not 
normally have a WBS. 
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Figure 3. Reference to naming system 
(From NASA Systems Engineering Handbook (5) , page 295) 


b. For convenience, the basis for the “Technology Scope Classification” shown on the axis of abscissas 
identified in Figure 2 and as presented in the Phase I Workshop was formulated by the Cost Analysis 
Division, and with an eye towards International Standards Organization (ISO) 27026 (22) . The ISO 
27026 issued a formative naming taxonomy and tree structure for cost breakdown structure in 
Appendix E of ISO 27026, which was used to help formulate the “Technology Scope Classification”. 

3 . Taxonomy of Technologies Between NASA Organizations 

Meaning: A naming system for technologies that is common between NASA organizations 

would help translate technologies and communication across organizations. Common naming systems 
would improve technology estimating and data management. 

Research 

Observation: NASA has multiple technology organizations and naming of systems between them 

was not observed to be standard. The common naming of items between organizations would help to 
retrieve information and cost and schedule data for technologies between different NASA organizations. 


4. Endpoint or Disposition of the Technology 

Meaning: The identification for what status the technology takes at its conclusion is important to 

understanding of the technology progression or if the technology has been dead-ended. The disposition of 
the technology is a clarification if the technology is to be tracked further or identified for how it can be 
progressed. 
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Research 

Observation: NASA does require technology projects to identify the end point or disposition of the 

technology. Identifying this information would help to track technology and illuminate the cost and 
schedule impact on future similar or same projects. 

The Phase I meeting confirmed that the focus of this research will include the scope for 
technologies with TRL range of 2 to 6 and that those technologies would be comprised of parts, materials, 
components, sub-systems or systems. Further, it was widely recognized within the NASA Cost 
Community that NASA currently has no cost estimating tool(s) for the scope of technologies within the 
range of TRL 2 to 6. Therefore, the focus of this Technology Estimating research project has been 
identified for technologies with TRL range of 2 to 6. 

In summary, multiple cost estimating processes were identified and evaluated with respect to 
the advantages and disadvantages presented , and considering the scope of technology addressed , 
parametric estimating offers the best option for use in this research to provide cost estimates for 
technologies in the TRL range of 2 to 6. However, based on the extent of retrospective data received 
during this phase of research, the analogy based methods will be used until the data can be evaluated 
further for Costing Estimating Relationship (CER) development and parametric analysis using those 
CERs. 

Technology Areas 

In 2004, the Steering Committee for NASA Technology Roadmaps; National Research 
Council of the National Academies published NASA Space Technology Roadmaps and Priorities: 
Restoring NASA f s Technological Edge and Paving the Way for a New Era in Space. On December 2012, 
NASA rolled out the NASA strategic Space Technology Investment Plan (1) . These reports identified 
NASA’s 14 technology roadmap to space technology. As a finding of the Phase I Workshop, four 
technology areas were identified to be investigated for Technology Estimating. It was resolved at the 
Phase I Workshop that the following technology areas, listed in Table 3, provided the highest opportunity 
for providing a large enough number of technology projects that could be investigated for use in the 
research. In addition, it was recognized that any attempt to include all 14 technology areas would be 
inextricably difficult within the one year term planned for this research effort. Further, the acquisition for 
data for all of the 14 technology areas would be a challenge within the time frame of this research. 


Table 3 

Phase I Workshop Results 

NASA Technology Roadmap Areas Identified to be Investigated in the Technology Estimating 
Research 

TA03 Space Power and Energy Storage 

TA04 Robotics, TeleRobotics, Autonomous Systems 

TA08 Science Instruments, Observatories, Sensor Systems 

TA12 Materials, Structures, Mechanical Systems, Manufacturing 


Parameters Used for Characterization of Technology 

Describing technology such that it can be adequately measured for use in Technology Estimating 
was one of the principle framework topics addressed in the Phase I Workshop. The Research Team 
engaged a three step process for development of parameters that would be used in characterizing the 
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technology. First, a literature search was conducted to determine those parameters that could best 
characterize NASA technology. The second step was to evaluate all the technology parameters in the 
Phase I Workshop that could best be applied for use in the cost estimating process and data collection. 

The third step included an evaluation of the parameters during the data collection phase, and for use in the 
Technology Cost and Schedule Estimating (TCASE) tool development. 


It was anticipated by the Research Team that the CERs will include multiple parameters that 
significantly influence the cost of technology development. These included parameters such as: 
Technology Readiness Levels (TRL), Key Performance Parameters (KPPs), Complexity, Research & 
Degree of Difficulty (R&D 3 ), Advanced Degree of Difficulty (AD 2 ), Implementation Readiness Level 
(IRL), etc. Initially, there were 13 parameters presented for consideration at the Phase I Workshop, see 
Figure 4. 


Qf 


Technology Estimating 


FRAMEWORK 

* What parameters DRIVE cost of Technology Development? 

* Short list best candidates for data collection effort? 

* Which ones have good data available? 

* Which ones could yield good data? //W 

* Many conceptual Parameters 



CANDIDATE 

PARAMETERS 


COST 
TRL 
IRL 
MRL 
FTE 
KPP 
RISK 
MASS 
AD? I R&D 3 
SCHEDULE 
COMPLEXITY 
BUSINESS ACUMEN 


Figure 4. Initial Phase I Workshop Parameters Presented for Consideration 


The advantages and disadvantages for prospective parameters presented for consideration in the 
Phase I Workshop and are provided in Table 4 for convenience, and also included are those parameters 
evaluated in the Phase II workshop. 
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Table 4 



Parameter - Advantages and Disadvantages 

1 

Parameter 

Advantages 

i 

Disadvantages 

I 

Technology Readiness 
Level 

i 

Experienced procedure; Assessment can affect Cost 
Estimate; Analysis using TRAT tool; Fast, iterative 
process easily repeated during development 

l 1 

Highly subjective; No defined process for defining 
TRL values; Characterize what drives the time to 
Technology Maturity; Difficult to measure; Risk to 
quality not fully developed; Identify future success? 

Key Performance 
Parameters 

Measurable performance goals that a given technology 
must attain to enable mission objectives; A significant 
advancement in the state-of-the-art 

Unique to each specific system, therefor application 
not well defined across multiple technologies 

Research and 

Compliments TRL by providing an accurate and effective 


Development Degree of 

assessment of the difficulty required for TRL 

Maturity of application & approach; Subjective? 

Difficulty 

advancement 


Manufacturing 
Readiness Level 

Adds analysis to determine manufacturability and 
identify related risks; Predicts whether or not we will be 
able to produce the product in the timeframe and at the 
rate desired with the desired quality; Identifies risks for a 
program office to work on. 

Subjective; No defined process for defining MRL 
values; Information must be dug out of 
manufacturer 

Criteria for comparisons must be established 

Advancement Degree of 
Difficulty 

Prior demonstration of implementation with AD 2 analysis 
tool 

Little demonstrated usage beyond AD 2 tool 


Defined schedule showing maturity increasing/adequate 

Technology development probability of failure is 

Schedule 

analysis and testing; High risk items, work around - 

similar to any project; Need defined WBS, 


Contingency development 

requirements, schedule, cost, etc. 

Integration Readiness 

Enables consistent comparison of the maturity of 

Insufficient for evaluating the developmental state, 

Level 

different integration points 

potential, or risk of a system 

System Readiness Level 
(SRL) 

Common platform for system development and 
technology insertion evaluation; Hierarchal view of 
technology insertion/system development assessment 

System readiness (capability, maturity) may be too 
complex concept; Calculated from other metrics 
rather than objectively measured, may yield 
inaccurate information and non-intuitive results 

Cost* 

Cost is fundamental to requirements for estimating 
new technologies. 

All costs may not be included due to shift in WBS. 

Estimate vs. Actual* 

Actual costs are preferred for parametric analysis. 

Estimates do not meet criteria for parametric 
analysis. 

Funding Sources* 

Provides understanding of possible program funding 

Multiple funding sources creates difficulties in 

impact to technology cost and schedule. 

tracking the technology costs 

Push vs. pull* 

Identifies the impetus for technology 

Application of classification is subjective 

Evolutionary vs. 
Revolutionary* 

Identifies the level of innovation 

Application of classification is subjective 

Technology Area* 

Segregates the technology into pre-defmed road map 
areas 

Cross referencing of technology areas can be 
confusing 
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Table 4 - Continued 



Parameter - Advantages and Disadvantages 

1 

Parameter 

Advantages 

i 

Disadvantages 

I 

Hardware vs. software* 

i 

Helps to illustrate the technology complexity 

I I 

Software can be developed as its own parameter 

Defining System 
Characteristics* 

Characterizes technology to a greater detail 

Difficult to apply consistent rationale 

Level of Effort* 

Defines the number of staff involved and potentially a 
good indicator of cost 

Students, contractor staff not well identified 

Unplanned Events* 

Helps to illustrate emergent costs 

Difficult data to acquire and information is 
currently not well documented 

Organization(s) 

Involved* 

Identifies the level of specialties required for a 
particular technology 

Difficult to establish as a cost driver 

Facilities and 
Infrastructure* 

Illustrates the basis for cost 

Difficult data to acquire and information is 
currently not well documented 

Team Experience* 

Exposes relation of technology complexity and ease of 
administration/accomplishment 

May be subjective 

Budget Constraints and 
Disruptions* 

Helps to illustrate emergent costs 

Difficult data to acquire and information is 
currently not well documented 


Note: * = Identified during and after Phase I Workshop 


Definition of Terms 

Initiation of this research required an assessment of the language terms that the Research Team 
used with one another in discussions and during the many briefings made throughout this research 
project. It was quickly recognized that a need existed to define terms, especially prior to the Phase I 
workshop. The construct of the definitions were intended to have a positive impact and to help elicit 
meaningful discussions amongst the workshop participants and provide a near community acceptance for 
how to communicate issues. In addition, the definitions were intended to be provided as a supplement in 
the data collection process. It is noteworthy that the definitions have not gained acceptance outside this 
research. The definitions were changed and modified multiple times over the course of the research to 
evolve into those definitions now used by the Research Team and they are presented in Appendix A. 

Technology Estimating 

For the purposes of this research, the phrase “technology estimating” implies the estimating of the 
cost and schedule requirements for the development of technologies. This differs from the interpretation 
of “technology estimating” used throughout most of the NASA community, for the broader technology 
estimating term often implies the analysis and estimation of the improved technical performance of a 
system or architecture due to the inclusion of a given technology. The usage here of technology 
estimating to mean the estimating of the cost and schedule for technology maturation is limited to the cost 
and schedule estimating communities. 
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Risk 


Risk may be defined as a measure for the loss of an asset or capability. Technology estimating 
provides the fundamental metrics that should help the technology manager reduce, eliminate or accept the 
risks pertaining to the development cost and schedule of a technology. The asset or capabilities that could 
be lost within technology development efforts include the success of the technology development effort 
itself. Once these risks are revealed or better understood, it will both enlighten and embolden the 
technology or technology portfolio manager and potentially help to support the foundation for partnership 
building. Partnerships that are founded on an understood cost and schedule risk will position the 
partnership to propel success in a technology’s research and its ultimate development. NASA has clearly 
indicated that partnerships for technology development, transfer and mutual benefit is a key objective in 
the Space Technology Program. (23) As an instrumental supporting role, technology estimating capability 
could help reduce risk by providing cost and schedule input data to decision support systems such as 
those identified by Smith et al., 2004 (24) and Weisbin et al., 2004, for complex technology portfolios. (25) 
Recognizing the full impact of cumulative risks in technologies within portfolio management could 
improve overall risk management and further the opportunity for success in the mission’s goal. 

Risk for technology projects has been addressed in the literature. (24) (25) (26) (27) Although it is recognized 
that technology risk is a driver to cost and schedule of technology ; due to the results of the Phase I 
Workshop , it was determined that there would be a limited or no data captured in a meaningful way 
for retrospectively reporting the risk data in the TRL< 6 projects . Therefore , risk as a parameter for use 
in analyzing individual technology projects would not be addressed in this research . 

Upon completion of the Phase I Workshop, and from input by technologist during the course of 
the research, a total of 20 parameters evolved and were then ultimately established for characterizing 
technologies and then used in the data collection. Figure 5 presents the 20 parameters identified. The 
parameters were identified into generalized classes to include: 

• Project characterization, 

• Project results, 

• Development metrics, 

• Project execution, and 

• Programmatic factors. 

Each classification was devised to help capture the overall technology project diversity, 
variability, and technical detail. These parameters have been defined in Appendix A. 
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Phase I Workshop Framework Parameters 


Project Characterization 

1 . Technology Area (TA) 

2. System Hierarchy 

3. Hardware vs. Software 

4. Push vs. Pull 

5. Evolutionary vs. Revolution 

6. Defining System Characteristics 

Project Results 

7. Key Performance Parameter (KPP) 

8. Level of Effort 

9. Cost 

10. Schedule 

Development Metrics 

1 1 . Technology Readiness Level (TRL) 

12. Research and Development Degree of 
Difficulty (R&D 3 ) 

1 3. Advancement Degree of Difficulty (AD 2 ) 


Project Execution 

14. Funding Source(s) 

15. Organization(s) Involved 

16. Facilities and Infrastructure 

17. Team Experience 

Programmatic Factors 

1 8. Estimates vs. Actuals 

1 9. Budget Constraints and Disruptions 

20. Unplanned Events 


• Collecting data on each of these 
parameters for all targeted projects is 
critical to enable the mathematical and 
statistical analysis needed to develop 
cost estimating relationships (CERs). 

• A Parameter Definition document is 
available for use by Technology 
Manager (subject matter expert). 

• Items in “Blue” Bold were added after 
the Phase I Workshop. 


Figure 5. Technology Estimating Parameters Identified in Phase I Workshop 
Used for Characterization of Technology and Data Collection 


Data 


Essential to this research project was the technology project data. Because limited historical data 
was believed to exist, or was anticipated to be available for the parameters which were expected to be 
significant drivers of technology development cost and schedule, a retrospective effort to gather historical 
data was required. Therefore, the focus of data for this research would be past projects that contained the 
best available information that could be readily found from online sources, embedded data in costing 
programs, or other Nasa Cost Analysis Division supported software. 

A significant focus in this research was to obtain historical project data , retrospectively 
generated , which would be used to help develop a key outcome of the research. This key outcome is a 
consensus in the cost and technology communities about those parameters which are determined to be 
key drivers for technology cost. Once this consensus is obtained , the parameters would be analyzed for 
development of cost estimating relationships. Once these relationships and parameters could be 
decided upon , then future collection of data related to these parameters can then be incorporated into 
ongoing databases and data collection efforts such as TechPort , One NASA Cost Engineering (ONCE) 
and Nasa Cost Instrument Model (NICM). Over time, existing CERs can be refined and new CERs can 
be developed as the amount of available data to conduct the requisite analysis expands. 
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1. Acquisition 


The acquisition of data for this research was broken down into three tasks for Technology 
Projects in the TRL range of 2 to 6 and the four technology roadmap TAs (3, 4, 8 and 12). These data 
acquisition tasks included: 

• Identification of a NASA community preference for which TAs, the NASA programs might 
prefer to be studied first; 

• Identification of the technology funding programs currently being administered by NASA as of 
2012; and then 

• Determine which of the available projects that could be identified by a survey given at the Phase I 
Workshop, and by survey of all data sources developed by exhaustive research. 

The first component to data acquisition was the Research Team’s task to determine if there 
existed an individual or most requested preference for which of TAs areas were to be studied and 
subsequently evaluated for different NASA programs or organizations. Based on a random survey of 
several technologists via informal interviews with different NASA programs, no preference was identified 
for one of the TAs was to be evaluated over another. 

The second component to the data acquisition was the identification of technology funding 
programs throughout NASA, and from these, it was necessary to identify which of those would 
potentially possess the necessary population of technical projects with data or information that might 
yield names of Technology Managers, and who could provide the requisite retrospective data. 
Approximately 42 NASA funding programs were identified initially for use by the Research Team for 
retrospective data generation by the Technology Managers. Table 5 lists the funding programs identified 
in the research to determine the availability of data. The list was derived from those programs identified 
and listed in TechPort. 
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Table 5 

Funding Programs Identified to Be Surveyed for Technology Estimating Data 

Number Centers & Facilities Programs 

1 Advanced Exploration Systems 

2 Astrophysics Research and Analysis 

3 Centennial Challenges 

4 Center Independent Research & Developments: GSFC IRAD 

5 Center Independent Research & Developments: JPL IRAD 

6 Center Independent Research & Developments: JSC IRAD 

7 Center Independent Research & Developments: KSC IRAD 

8 Center Independent Research & Developments: LaRC IRAD 

9 Center Innovation Fund 

10 Center Innovation Fund: ARC CIF 

1 1 Center Innovation Fund: DFRC CIF 

12 Center Innovation Fund: GRC CIF 

13 Center Innovation Fund: GSFC CIF 

14 Center Innovation Fund: JPL CIF 

15 Center Innovation Fund: JSC CIF 

16 Center Innovation Fund: KSC CIF 

17 Center Innovation Fund: LaRC CIF 

18 Center Innovation Fund: MSFC CIF 

19 Center Innovation Fund: SSC CIF 

20 Discovery Program 

21 Earth Science Technology Office 

22 Exoplanet Exploration 

23 Flight Opportunities Program 

24 Fundamental Aeronautics Program 

25 Game Changing Development 

26 Ground Systems Development and Operations 

27 HELIOS 

28 Mars Exploration 

29 NASA Electronic Parts and Packaging Program 

30 NASA Innovative Advanced Concepts 

3 1 OSMA Software Assurance Research Program 

32 Planetary Science Research 

33 Planetary Science Technology Program 

34 SBIR/STTR Programs 

35 Small Spacecraft Technology Program 

36 Space Life and Physical Sciences Research and Applications 

37 Space Technology Research Grants 

38 Technology Demonstration Missions 

39 Phase I SBIR 

40 Phase II SBIR 

41 Phase III SBIR 

42 Constellation ETDP (Exploration Technology Development Program) 
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The third component for data acquisition was an exhaustive survey of the available technology 
projects that may be available for use in the research. To help identify possible technology projects with a 
potentially rich data to help satisfy the retrospective data generation by the Technology Managers, a 
survey was conducted at the Phase I Workshop. The survey included a written document (form) which 
was developed and provided to the Phase I Workshop participants. The form is included in Appendix B. 
The information obtained from the survey during the Phase I workshop was promising, however the 
resulting number of individual Technology Projects revealed as a result of this survey were ultimately 
found to be limited. In addition to this survey at the Phase I Workshop, a listing of 76 prospective data 
sources was developed by the Research Team and is included in Appendix C. This listing was based on 
research of NASA programs, literature search, and from cost estimating models common to NASA and 
commercial industry typically used for TRL > 6 missions and projects. 

To help acquire the needed data for the research, a Data Collection Sheet was prepared by the 
Research Team and consisted of an Excel® spreadsheet in three printable sheets and included in 
Appendix D. In preparation of the collection sheet format, the Research Team generally followed the 
NASA’s TechPort for its sequence of information as presented on the web page. In addition, the data 
collection effort generally followed the principles as outlined by Meisel, 1988, who identified three key 
requirements for obtaining the correct model input data: (1) finding the right experts, (2) asking the right 
questions, and (3) transforming the question responses correctly into data compatible with the model 
input format. (28) 

The format of the data collection sheet along with the instructions for preparing answers was 
discussed and modified at least four times by the Research Team. After resolving many concerns, the data 
collection sheet was then used in at least three separate and individual interviews with Technology 
Managers for preliminary testing. The format of the data collection sheet consisted of pull down menus 
where possible so as to aid the Technology Manager in making a response. File extension naming of the 
data collection sheets also provided very helpful in identifying the correct projects with the respective 
Technology Manager and accountability. The data collection sheets were also made suitable for data 
retrieval into an Integrated Technology Project Database which was used as a data source to the 
Technology Cost and Schedule Estimating (TCASE) tool. 

Overall, data collection followed guidelines as identified by established methodologies (29)(30) 
including structure in the document, pre-testing with sample technologist, and including wording or 
sequence to help reduce bias. 

2. Sources of Data 

The existing data sources identified in the first phase of the research were compiled and 
investigated in detail. Those with reliable cost and schedule information were selected for use as 
supporting data for the TCASE framework. The specific data sources are identified below, along with the 
year the data source was produced, and the number of entries (i.e. technology data projects) contained in 
each. Included are those 3,178 data found as listed in Table 5, by SpaceWorks, 2012 (31) 
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Table 5 




Data Sources Used in this Research (31) 



Short Name 

Description 

Year 

Entries 

ETDP 

ESTO 

SBIR 

GCD 

ESMD 

NASA Exploration Technology Development Program 
NASA Earth Science Technology Office 
Current NASA SBIR Technologies 

Game Changing Development Program Office Technology 
Projects 

NASA Exploration Systems Mission Directorate ESAS 
Technologies 

ATLAS (Advanced Technology Lifecycle Analysis System) 
Tauri Group research into External Government Technologies 
(EGT within the MATCH relational database) 

2012 

2012 

2012 

2012 

2005 

42 

138 

51 

35 

304 

Tech Tool Box 
Ext Tech Data 

2002 

2012 

6 

28 

Cx ETDP 

Constellation Exploration Technology Development Program 

2007 

19 

NTID 

NASA Tech Inventory Database 

2004 

991 

Hist SBIR STTR 

Historical SBIR and STTR data 

2012 

1191 

RLV Tech DB 

NASA RLV Technology Database 

1993 

64 

HEOMD 
TechDev Team 

Estimates of HEOMD needed technology developments 

2012 

78 

ETDP from 
MATCH 

Mapping Applicable Technology To Exploration Challenges 

2007 

21 

External from 
MATCH 

Mapping Applicable Technology To Exploration Challenges 

2007 

192 

JSS TBCC 

NASA/Air Force Joint Systems Study TBCC Technologies 

2010 

18 


SpaceWorks, 2013, provided a frequency analysis of 643 technology projects investigated for all 
technology areas, Figure 6 to identify if the source of data were rich with the 18 parameters of interest, 
and then only for the four selected technology areas were presented in Figure 7. 
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Parameter Frequency Analysis 


Objectives 

* Test for significant relationships between collected parameters and 
development cost and schedule. 

* Assess the relative value of each current parameter and recommend 
a shortlist of parameters for future collection efforts. 


Analysis Results (from 643 records) 

* Caveat Data must be available in order to quantify relationships. 
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Mote: Continued retrospective data 
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Figure 6. Data Frequency - 20 Parameters 
(Figure from SpaceWorks, (31) 2013) 
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Figure 7. Data Richness - Data in four Technology Areas (3, 4, 8 & 12) and 18 Parameters 

(Figure from SpaceWorks, (31) 2013) 

As indicated in Figures 6 and 7, many parameters were found to be absent in the historical data 
for all the data sources investigated and specifically these included: team experience, system hierarchy, 
hardware vs. software, R&D 3 , AD 2 , push vs. pull, evolutionary vs. revolutionary, and unplanned events. 
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Upon completion of an evaluation by the Research Team of all the data sources, it was revealed 
that the SBIR Phase III, ETDP, ESTO and GCDPO programs retained the highest potential sources of 
data to be used for retrospective data generation to be used in the research and for evaluation of the 20 
parameters for possible CER development. This was due to the nature that much of the project 
information was available on-line or as a downloaded document with cost and schedule data furnished to 
the Research Team. These four sources of data for these projects were identified in Table 6. These 
specific programs were found to have many of the data and parameters of interest to the research which 
included published: cost and schedule data, TRL, project descriptions, technology area identified, and the 
names of Principal Investigators or Project Managers and contributor contractors. Initially, 229 
technology projects were identified individually as containing potentially high value data. Multiple 
contacts were made with technologists where the Technology Managers reside to request assistance for 
their support to supply the needed data. 


Table 6 

Sources of Technology Projects TRL 2 to 6 Used for Evaluation of the 20 Parameters 
SBIR Phase III, EDTP, ESTO and GCPDO Programs 

Funding 

Program 

Media 

Point of Contact 

Notes 

SBIR Phase II 
& III 

List from 

Electronic 

Handbook 

Provided by Mederos, L. and Jahns, G. 
C. 

Cost and schedule 
data provided 

ETDP 

CD 

Fitzgerald, J. and Law, R. C. provided 
duplicate copy of the ETDP information. 
Formal copy of the data was included in 
Windchill™ as indicated by Hall, K. L. 
but not readily accessible. 

Partial cost and 
schedule data 
available. ETDP 
data was headed to 
National Archives. 

ESTO 

On Line 

On Line at 

http://www.estotechnologv.us/techportfo 
lio/ and Komar, G. J. 

Cost and Schedule 
data provided 
separately. 

Game 

Changing 

Development 

Program 

Office 

NX 

Koca, M. E. and Brooks, C. E. 

Cost and schedule 
data available. 


Beginning May 21, 2013, 178 of the Technology Managers for 229 targeted Technology Projects 
were provided a data collection sheet to be used for retrospective data generation. The balance of the 5 1 
technology projects were found to have Technology Managers who were no longer associated with the 
project or not found, or no records were found to be available to make for a viable data collection 
response making these data of no value. As of June 20, 2013, 37 data collection sheets were prepared by 
the Technology Managers and returned to the Research Team. A copy for each of the electronic data 
collection sheets for responses received from the Technology Manager are included in Appendix D. 
Where telephone contact with the Technology Managers were necessary to aid in the completion of the 
data collection sheets, GAO structured interviewing techniques and guidelines (28) were implemented to 
help provide explanations that were uniform and to provide clear explanations or instructions to avoid 
disruption of data quality, avoid introduction of bias, or to not interfere with the cognitive decision 
process of the Technology Manager. (28 29) 
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An informal survey of the Technology Managers, who provided their input to the data collection 
sheet, or their facilitators related to the data collection activity generally, indicated the following: 

• “This is a very important process, and we need it”, 

• “Insufficient time to accomplish this data collection effort with all the other work being 
done”, 

• “It takes from one to two hours to complete the data collection sheet for a project”, 

• “The technology project is old and I cannot remember, or files not available”, 

• “The project is 10 years old”, and 

• “A WBS will be needed to complete the data collection sheet”. 

Data collection from established sources in this research proved exceedingly difficult and the 
uniform documentation of the data in a useable format overall was found not well established. A notable 
exception, the data acquired from the SBIR Electronic Handbook, was superior to this research in that it 
contained a systematic categorization of the many parameters of interest and the data were presented on 
an Excel® spreadsheet. The Game Changing Development Program Office data were found to contain the 
highest level of parameters of interest to the Research Team and the technology project data were found 
archived as slides in Microsoft™ Power Point® on LaRC’s NX on-line project archival system. The 
ETDP data was found to be located on NASA’s Windchill® system and designated to be headed for the 
National Archives. An attempt was made to intercept it for use in this research and fortunately, a copy of 
the ETDP data was made available to the Research Team before its’ archival. The ESTO project 
information was rich in data and found on an on-line web site quad sheets in Adobe™ PDF® format. 
These ESTO project data were then converted to Microsoft Word® documents for extraction to the 
Integrated Data Base. 

Appendix B of the Parametric Handbook identifies that data should be collected for all 
technology projects centrally and maintained in a manner that provides a complete audit trail with 
dates , so that cost can be adjusted for inflation. (2I) A good example for how project metrics are 
documented for large mission level projects was NASA’s ONCE and CADRe. It was unfortunate 
however that the ONCE/CADRe system did not afford a suitable data base for this research. ONCE and 
CADRe is supported by requirements included with NASA policy for NASA Procedural Requirements 
(NPR) 7120.5 projects in excess of $250 million. It is recognized that Technology projects of TRL range 
of 2 to 6 are below that dollar threshold and may be managed at a different level under multiple funding 
programs. Each of these lower TRL technology projects has been founded on programs that have different 
documentation requirements and none were required to be centrally located in a data base. It was also 
observed that Technology Projects with TRL < 6 in this research do not normally have a work breakdown 
structure in which data can be acquired. Further, programmatic information related to skill of the 
technology team, infrastructure requirements, delay (s) to the project, significant change(s) in the project, 
and tracking the project were obscure. 


3 . Pre-Completed Data 

Contact with several NASA organizations revealed that the time for generation of the 
retrospective project data into the data collection sheet by the Technology Manager was a perceived 
burden to their time applied to their required duties. In an effort to help reduce this time burden, the 
Research Team attempted to obtain total project costs and the beginning and end dates for the project 
from the respective funding organization. In addition, an effort was made to retrieve the technical paper(s) 
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published for 82 technical projects. The technical paper(s), where found were researched to determine if 
they contained any one of the 20 project parameters to be evaluated for characterization of the 
technology. The information contained in the technical paper(s) was then translated back to the project 
Data Collection sheets along with the cost and schedule information as provided. The Technology 
managers were then asked to back check the pre-completed data collection sheet for accuracy. 

It is important to note that the NASA Aeronautics and Space Database was the primary research 
source for investigating technical publications related to the technology project along with the LaRC 
OCIO Technical Library resources and E-Databases. At times, this effort was augmented by courtesy use 
of the Old Dominion University Library on-line resources. 

The following was learned from this pre-completion effort for the data collection sheet: 

• The technology presentation for each project was researched. Some technology projects only had a 
power point presentation associated with the technology project as if they had been given to a 
professional technical group or symposium. The value for obtaining parameters from this source was 
found to be of low or no/questionable data value, and the data collection sheet was not pre-completed. 

• The technology paper for each project was researched. Technology projects at times had one or more 
technical papers associated with the technology project. Most papers were found to consistently 
contain rich information with respect to the 20 parameters of interest. The value for obtaining data for 
the estimating parameters and retrospectively generating the data into the data collection sheet was 
found to be of high to medium data value. 

• During the research, the technical papers did not always possess the same title as the funded research 
project title. 

• Tracking the technical paper to the parent funded research project title was at all times not possible. 
The funding contract number did not appear in the technical paper. 

• A question was identified that could not be resolved within the scope of this effort to understand if a 
series or successive technical papers were connected to the parent funding contract. Currently, 
progression of the technology could only be tracked by the technical paper’s publication date. It was 
unresolved in this limited time to determine if the primary technology branched into other technology 
applications or Technology Areas. 

• The TRL progression and cost per year was input as an estimate to the pre-completed Data Collection 
sheets by the Research Team as a straight-line interpolation, 

• Review of the technical papers did not reveal if a cost was associated with use of a facility for the 
project. When costs are reported in pre-completed data collection sheets for use of facility, they were 
estimated by the Research Team. 

• Review of the technical papers did not reveal how many Full Time Employees (FTEs) were included 
for a technology. The numbers of FTEs were estimated on the Data Collection Sheets by the Research 
Team based on the number of technical paper contributors. 
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• Typically, the pre -prepared data collection sheet, if not checked or confirmed by the by the 
Technology Manager, was ultimately identified to be of “no/questionable value” data and was not 
used. 

• Between the initiation of data collection in March, and through June 2013, approximately 17% of the 
Technology Managers responded even though the Data Collection Sheets were pre-completed for 
their review and back-checking. By Mid-September 2013, the response rate was approximately 31%. 
The participation by the Technology Managers was gained from direct contact via emails and phone 
calls. 

4. Data Quality 

Data to be used in the research must be satisfactory for use in the development of the cost 
estimating relationships that are included in the parametric analysis. Multiple screening criteria were 
established in a written document titled Technology Estimating Data Quality and Screening Methodology 
and the data was checked for general accuracy. The Data Quality and Screening Methodology document 
has been included in Appendix E. 


5. Parameters Evaluation and Screening 

During the Phase I workshop, 20 parameters (See Figure 4) were established for the 
characterization of Technology projects. These parameters were classified into categories for Project 
characterization and results, development metrics, project execution, and programmatic factors. 
Recognizing the variability of technology, and the goal of determining which one of these parameters 
drive cost an effort was made to understand and confirm what it is we are trying to accomplish. The 
Research Team focused on a path to understand what the fundamental drivers to technology cost and 
schedule were and how best do we help assign these drivers a place in the parametric analysis to provide 
an estimate and schedule. 

The parameters were evaluated to determine if they retained either a qualitative or quantifiable 
objective and the results are provided in Table 7. The qualitative and quantitative aspect of each 
parameter was analyzed with a discussion indicated. 
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Technology C 

Table 7 

haracterization Parameters - Analysis 

Project 

Characterization 

Parameter Value 

Metric 

(Quantifiable 

or 

Qualitative) 

Discussion 

Technology Area (TA) 

1 through 15 

Quantifiable 

Easy parameter to assess for first level TA, however the secondary, tertiary 
levels of TA were at times difficult to apply. 

System Hierarchy 

Hardware/Material , 
Component/Part, Assembly, 
Sub-system, System 

Qualitative 

The hierarchy was easy to apply and helps to classify the technology. 

Hardware versus 
Software 

All hardware/No software, High 
hardware/Low software, 
Medium hardware/Medium 
software, Low hardware/High 
software, No hardware/ All 
software 

Qualitative 

The amount of hardware versus software was found difficult to quantify with 
great accuracy. The level of programming hours, lines of code, extent of 
computer applications must be addressed in order to determine its total cost 
driving implications to the Technology. Parameter should be expanded to 
include software as a separate parameter. 

Push versus Pull 

Either 

Qualitative 

The determination of the intent whether a project was push versus pull could 
not be fully discerned from the data. 

Evolutionary versus 
Revolutionary 

Either 

Qualitative 

The determination of the intent whether a project was evolutionary versus 
revolutionary could not be fully discerned from the data. 

Defining System 
Characteristics 

Description of project in terms 
of length, mass, volume, power, 
etc. 

Quantifiable 

Parameter was a significant contributor to the analysis as a means to describe 
the technology however at the level of limited data for a specific TA. 

Project Results | 

Key performance 
Parameter (KPP) 

Description of project 
performance or distinct 
capability 

Quantifiable 

The parameter was a significant contributor to the analysis as a means to 
describe the technology however at the level of limited data for a specific TA. 

Level of Effort 

Full time employees 

Quantifiable 

The parameter was considered a significant contributor to the research. 

Cost 

Total cost in dollars 

Quantifiable 

The parameter was considered a significant contributor to the research. 

Schedule 

Total project duration in years 

Quantifiable 

The parameter was considered a significant contributor to the research. 

Development Metrics ( 

Technology Readiness 
Level (TRL) 

1 through 9 

Qualitative 

The parameter is a significant contributor to the analysis however the TRL 
determination may is subjective. 

Research & Development 
Degree of Difficulty 
(R&D 3 ) 

1 through 5 

Qualitative 

The parameter was found difficult to apply. 

Advancement Degree of 
Difficulty (AD 2 ) 

1 through 9 

Qualitative 

The parameter was found difficult to apply. 

Project Execution | 

Funding Sources 

Number and identification 

Qualitative 

The parameter was found easy to apply. 

Organizations Involved 

Identification of NASA 
organizations, Universities, 
contractors, etc. 

Qualitative 

The parameter was found easy to apply. 

Facilities and 
Infrastructure 

Description of laboratories, 
aircraft, or other special facilities 
required. 

Quantifiable 

The parameter was found easy to apply. 

Team Experience 

Little or no team experience, 
Experience conducting research 
but little or no experience with 
specific technology, Experience 
with research of this specific 
technology, Experience with this 
specific technology and 
innovative team 

Quantifiable 

The parameter was found easy to apply 

Programmatic Factors 

Estimates versus Actuals 

Either 

Quantifiable 

The parameter was found easy to apply. 

Budget Constraints and 
Disruptions 

Description of any budget 
constraints or disruptions which 
may have influenced the project 

Qualitative 

The parameter was found easy to apply. 

Unplanned Events 

Description of any events which 
may have influenced the project 

Qualitative 

The parameter was found easy to apply. 


An analysis was conducted of the 20 parameters further using a weighting scheme and the results 
are presented in Figures 8a and 8b. 
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Figure 8a. Parameter Analysis Using a Rating Scheme 
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Figure 8b. Parameter Analysis Using a Rating Scheme 


Each parameter was individually evaluated and documented with an evaluation narrative and 
included in Appendix F. The evaluation conducted used the premise that a schedule and cost driver was a 
parameter that tends to increase or decrease schedule/cost or is highly correlated with schedule/cost, and 
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to qualify as a cost driver, a parameter must be a significant predictor of schedule/cost at least in the 
statistical sense. The following is a detailed summary of the parameter evaluation: 

A statistical analysis was conducted to evaluate the parameters and the results are included in 
Figure 9. 


Analysis of Variance (ANOVA) Results 


Parameter 

Likelihood of Relationship between 
Parameter and Cost? 

Comments 

Technology Area (TA) 

Significant (p-vaiue * o oo> 

Samples: 605 historical + 38 retrospective 


Total Cost 

- 

This is an output (predicted) parameter 

Schedule (duration) 

- 

This is an output (predicted) parameter 

TRL (i.e. TRL Start^End) 

Significant (p-vaiue*o.oo) 

Samples: 485 historical ♦ 37 retrospective 


Funding Source(s) 

- 

Insufficient commonality/uniformity within 
data to conduct test 

Level of Effort 

- 

Not an independent parameter 

System Hierarchy Level 

Significant (p-vaiue = o 03) 

Samples: 213 augmented + 26 retrospective 

Further evaluation is required 

Cost Type (Estimates vs. 
Actuals) 

- 

Intended for classification of data only; Not 
predictive 

KPP 

- 

Insufficient commonality/uniformity within 
data to conduct test 

Performing Organization(s) 

Insignificant <p-vaiuo = o 43) 

Samples: 605 historical ♦ 38 retrospective 



Note : It is common practice to look for a p-value <= 0.05 (i.e. 5%) before declaring that there is a 
significant underlying relationship between groups of data. 


Figure 9. Analysis of Variance Results 


With the addition of retrospective data, additional analysis is expected to be conducted to 
determine the significance of the parameters to be evaluated further. 


Based on the analysis conducted, the parameters were screened and the recommended parameters 
were then identified in the Phase II Workshop. Based on the workshop results, the following parameters 
are recommended to be used for Technology Estimating, Figure 10, and those items identified with a 
“star” are already included in TechPort. 
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Preliminary Recommended Parameters 

Project Characterization 

Project Execution 

^ 1. Technology Area (TA) 

14. Funding Source(s) 

Z. System Hierarchy 

15. Organization^) Involved 

3. Hardware vs. Software 

1 6. Facilities and Infrastructure 

4 . Push vs. Pull 

17. Team Experience 

4 Evolutionary vs. Revolutionary 


S. Defining System Characteristics 

Proqrammatic Factors 


— Estimates vs Actuals 

Project Results 

— Budget Constraints and Disrupt 

ions * 

7. Key Performance Parameter (KPP) 

20. Unplanned Events 

8. Level of Effort 


9. Cost 


10. Schedule 

20 Parameters 

Development Metrics 

Reduced to 15 or 16 

11. Technology Readiness Level (TRL) 

= Included in TechPort 


12 Research and Development Degree of 

* Combine with Unplanned Events 


Difficulty (R&D 3 ) 



13. Advancement Degree of Difficulty (AD 2 ) 

Italics - Status To Be Determined 



Figure 10. Preliminary Recommended Parameters 


Several parameters were identified to be evaluated further. The parameter hardware versus 
software has been identified to have the hardware term eliminated and the software parameter term could 
include: 

1 . Brief description of software and its use, 

2. Source lines of code, 

3 . Function point count, 

4. Object class count, 

5. Number of screens, 

6. Language, 

7. Application type, 

8. Build environment, 

9. Target environment, 

10. Reliability, Maintainability, 

11. Degree of reuse or COTS, 

12. Number FTEs, and 

13. Team experience. 
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Estimating Tool 


Development of a tool to aid in the preparation of an estimate must include a sober consideration 
for its intended accuracy. Oberlander et al., 2001, identified that the accuracy of an estimate is measured 
by how well the estimated cost compares to the actual total installed cost. Oberlander’ s research identified 
that the accuracy of an early estimate depends on four determinants: (1) who was involved in preparing 
the estimate: (2) how the estimate was prepared; (3) what was known about the project; and (4) other 
factors considered while preparing the estimate. Further, his research team defined the estimate as the 
point in time when the estimate was performed, i.e. estimate that has been prepared from inception of the 
project up to and including funding approval. (9) The Technology Estimating research has determined 
that the estimating process presented herein is initially conceived as a predictive estimate performed at 
the inception of the technology , and based on the level of confidence indicated with the number of 
similar projects included and with reference to their respective analogies of projects . Ultimately , it is 
desired that the process will lead to development of Cost Estimating Relationships and Parametric 
analysis to be developed later. 

The credibility of an estimate is a highly desirable but intangible characteristic that is often touted 
but can only be achieved through demonstrated performance or through verification or validation. 
Determining the credibility of an analogy based or parametric model whether (1) to verify the cost of a 
program for a higher command, or (2) to develop a cost estimate, or (3) to evaluate a cost proposal 
requires the same type of information. The responsible person, or group(s), accepting the output of the 
model must feel comfortable with the information being provided. Fad, et al. 1988, indicated that this can 
only come from a moderate understanding the model being used, information pertaining to its technical 
validity, confidence in the people using it, and the track record of both. (20) Colmer, et al., 1999, 
recognized that although cost estimation techniques can make forecasts using very little data, it is a 
mistake to place great reliance on these estimates without a firm grasp of the estimate limitations, 
assumptions, and risks. All too often a number (or cost estimate) created has a life of its own and 
continues to mislead. It should be borne in mind that new technologies have been developed before. 
Experience on how this affected cost is still relevant, especially as it is likely that some parts of the 
product will be little changed by these developments. A sufficiently long term historical perspective and a 
broad view of industry may well reveal trends and analogies of value to the estimator. (32) It is recognized 
in this research that development of a technology estimating process and tool will evolve to a higher 
levels of credibility once the commitment in this research continues and the level and number of data 
required on which the technology estimates are based becomes increasingly established. The high level 
of credibility anticipated to be required for users will continue as a function of the long term and 
perpetual commitment to data acquisition and its maintenance. 

The TCASE tool was developed in Microsoft Excel® from the modeling architecture functional 
diagram developed and presented in Figure 1 1 . The tool allows the user to define a new technology 
project for the purpose of cost and schedule estimation by entering inputs for the different estimating 
parameters identified during the study. To familiarize the user with the tool, a User’s Manual is included 
in Appendix H. Based on these inputs; the tool presents analogous historical technology projects from the 
database of collected project data. From these analogies, a statistical distribution of cost and schedule 
predictions can be generated. 
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TCASE Functional Diagram 



Figure 11. Architecture of the Technology Estimating Tool TCASE 
(Figure from SpaceWorks, (33) 2013) 

The front end of the TCASE tool was updated to reflect the development of the analytical 
framework. The Project Inputs section reflects the latest technology cost and schedule estimating 
parameters, and the analogies section was expanded to include the parameters associated with the 
analogous technology projects. The front end is shown on Figures 12a and 12b. 
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Figure 12a. Front End View of TCASE 
(Figure from SpaceWorks Phase II Workshop, (33) 2013) 



TCASE Screenshots 


User Interface - Details 


Search Criteria 

1 


Match search term 
exactly (optional] 





Clearly indicates number 
of data records 
(analogies) returned by 
your search 


Mfdtoi $MJ.6}6 





Box plots indicate 
center and spread 
of analogy results 



Breakdown plots 
provide contextual 
information about 
analogy results 



■ 


l. 


Figure 12b. Front End View of TCASE 
(Figure from SpaceWorks Phase II Workshop, (33> 2013) 
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Importing and Exporting Technology Project Data Collection Worksheets was addressed by the 
Research Team in consideration of current and future applications for updating the TCASE with data. The 
data collection sheet can be used by the technologists for recording the parameters associated with their 
technology projects. In order to easily incorporate new data into the existing dataset, the processes of 
importing data from the datasheets to the database, and exporting data from the database to the datasheets, 
were automated through a set of Excel® VBA macros. 

To import data from a completed data collection sheet, the VBA macros parse the datasheet, 
collect the parameter data from the different fields, and store this data in memory. The macros then print 
the data into a new row of the database, with each field from the datasheet corresponding to a column of 
data in the database. To export data from the database, this process is reversed. The macros read the 
parameter data from a single row in the database, and print this data into the appropriate fields in the 
datasheet. The macros are designed to import and export multiple technology projects simultaneously. To 
import multiple datasheets, all of the datasheet files are placed in the same folder; the macros open every 
datasheet identified in the folder and print the parameter data to multiple rows in the database. Conversely 
if multiple projects are to be exported from the database, the macros read multiple rows from the 
database, create a new datasheet file for each row, and print the parameter data for the projects into their 
associated datasheets. 

Calibration, verification, validation and accreditation of the model were addressed by the 
Research Team. As this research will introduce the viability of the technology estimating process and 
TCASE tool for only four TAs, full accreditation was not anticipated to be fulfilled. Verification and 
validation of the TCASE however will be accomplished by continued investigation of case studies. 
Documentation on model verification and validation is usually critical in convincing users of the 
“correctness” of a model and its results, and should be included in the simulation model documentation. 
Calibration is the process of adjusting a model’s parametric values to reflect an organization's cost and 
product history. The calibration process converts a general model into one developed exclusively for the 
organization’s application. Validation is the process or act of demonstrating a model’s ability to function 
as a credible forward estimating tool or system. Validation is performed for all parametric estimating 
techniques (e.g., CERs, models). (18) 

Verification and validation steps generally considered and applied the following after Sargent, 
2000 to TCASE: 

1 . Apply a test case as a study to identify that the cost and schedule estimates are at least bracketed 
within the range of costs for analogous projects. 

2. The amount of accuracy required of the model’s output variables for the model’s intended 
application is dependent upon the input data. 

3. Test, wherever possible, the assumptions and theories underlying the model. 

4. In each model iteration, perform at least face validity analysis of the conceptual model. 

5. In each model iteration, at least explore the model’s behavior using the computerized model. 

6. In at least the last model iteration, make comparisons, if possible, between the model and system 
behavior (output) data for several sets of experimental conditions. 

7. Develop validation documentation for inclusion in the simulation model documentation. 

8. Over a period of time, a schedule model upgrades will be ultimately developed for periodic 
review of the model’s validity. (34) 

The TCASE analytics was based on work by Opricovic et al. (2006) VIKOR method for multi- 
criteria decision making analysis which is an effective tool in multi-criteria decision making, particularly 
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in situations where the decision maker is not able, or does not know to express his/her preference at the 
beginning of system design. The obtained compromise solution could be accepted by the decision makers 
because it provides a maximum group utility of the “majority”, and a minimum individual regret of the 
“opponent”. The compromise solutions could be the base for negotiation, involving the decision makers’ 
preference by criteria weights. (35) 

The Exact Match parameter can be set to either yes or no. If set to yes, then that particular search 
term will be treated as a filter, meaning that any project in the database that does not contain that search 
term in the specified field will not be returned as a viable result. 

The Weighting parameter can be used to establish the relative importance of each search term 
(e.g. assigning a weighting =1.0 for Title Keywords and a weighting of 2.0 for Primary TA would 
indicate that matching the Primary TA search term is twice as important as matching the Title Keywords). 

The two parameters listed under Search Settings (Minimum Score for Analogy Matching and 
Maximum Number of Analogies in Results) are optional parameters that can be used to limit the number 
of results returned by a particular search. Lowering the minimum score or raising the maximum number 
of analogies will increase the number of results returned, and vice versa. 


Findings 

The following findings were provided: 

1 . A basic assumption taken in this research was that all technologies have inherent 
variability . 

2. Parametric estimating provides the best possible solution for satisfying this technology 
estimating research . 

3. The technologies of interest in this research study were TRL of range 2 to 6 . 

4 . Tracking technology in its early progressive stages along the path of development or where 
the early technology has branched to other Technology Roadmap Areas (TAs) would be 
paramount as a significant building block to Technology Estimating. 

5. Multiple cost estimating processes were identified and evaluated with respect to the 
advantages and disadvantages presented , and considering the scope of technology 
addressed , parametric estimating offers the best option for use in this research to provide 
cost estimates for technologies in the TRL range of 2 to 6. However, based on the extent of 
retrospective data received during this phase of research, the analogy based methods will 
be used until the data can be evaluated further for Costing Estimating Relationship (CER) 
development and parametric analysis using those CERs. 

6. Four technology areas were identified to be investigated for Technology Estimating. 

7. Although it was recognized that technology risk is a driver to technology cost and schedule, 
due to the results of the Phase I Workshop it was determined that there would be a limited 
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or no data captured in a meaningful way for retrospectively reporting the risk data in 
completed projects . Therefore , risk as a parameter for use in analyzing individual 
technology projects would not be addressed in this research . 

8. A significant focus in this research was to obtain historical project data , retrospectively 
generated , which would be used to help develop a key outcome of the research . This key 
outcome is a consensus in the cost and technology communities about those parameters 
which are determined to be key drivers for technology cost. Once this consensus is 
obtained , the parameters would be analyzed for development of cost estimating 
relationships. Once these relationships and parameters could be decided upon , then future 
collection of data related to these parameters can then be incorporated into ongoing 
databases and data collection efforts such as TechPort, One NASA Cost Engineering 
(ONCE) andNasa Cost Instrument Model (NICM). 

9. Data should be collected for all technology projects centrally and maintained in a manner 
that provides a complete audit trail with dates, so that cost can be adjusted for inflation. 

10. The Technology Estimating research has determined that the estimating process presented 
herein is initially conceived as a predictive estimate performed at the inception of the 
technology, and based on the level of confidence indicated with the number of similar 
projects included and with reference to their respective analogies of projects. Ultimately, it 
was desired that the process will lead to development of Cost Estimating Relationships and 
Parametric analysis to be developed later. 

11. It was recognized in this research that development of a technology estimating process and 
tool will evolve to a higher levels of credibility once the commitment in this research 
continues and the level and number of data required on which the technology estimates are 
based becomes increasingly established. The high level of credibility anticipated to be 
required for users will continue as a function of the long term and perpetual commitment 
to data acquisition and its maintenance. 

12. The Research Team identified gaps during the conduct of the research. These included: 

a. Cost and schedule data for some Technology Projects were not readily available in an 
accessible open source or not available at all. 

b. Research papers for the Technology project were attempted to be used to help satisfy 
Technology Estimating data collection effort. Technology Projects did not always have 
a research paper. 

c. Verifiable means to track a technical paper back to the funded research project was not 
found. 

d. Where multiple technical papers were produced, no means existed to verifiably track to 
the funded research project. 

e. Since this research was conducted primarily for completed projects, not all of the 
completed projects were assigned to a NASA Technology Roadmap Area. 

f. Tracking low TRL projects through the system to a higher level was not possible. 
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g. Although TechPort was released during the course of the research , TechPort data 
could not be used exclusively as it understandably had not been backfilled with 
completed projects . 

h . The SAP system was initiated on or about 2003. Cost data for Technology Projects 
before 2003 is not readily available. 

i. TechPort could be optimized further to include additional parameters data , requiring a 
log in to view the specific parameter. 

13. Improvements to the TCASE tool may include: 

a. An investigation should be made to determine if it would be useful to have the option of 
selecting one analogy and pulling up all of its available data into one tab/sheet rather 
than using its ID to search the database. The Research Team will investigate to 
determine if this data base query capability for one analogy can be provided in future 
version of the tool. 

b. Annotating the percentage as a score of parameter “hits” as relevance , and/or 

c. Listing the parameters with a ranking for each or say using a top to bottom score for 
each parameter in the analogous project. 

d. System Hierarchy for the project as the system level down approach to the 
Hardware/Material level was addressed however ; some of the data in the TCASE model 
appears inconsistent with this breakout. Some projects seem to include much more 
cost beyond just an integrated system/spacecraft level. The TCASE should be refined 
further with respect to the data according to the system Hierarchy. 

e. The Research Team should identify if there is a way to identify the technology data 
that’s been validated for cost , schedule , TRL, and other parameters. 

f. The Research Team will verify that the tool data base does not have duplicates. 

g. The Research Team will verify that TRLs listed in the data base appear correct and 
that lower TRL at the end of the project are consistent with the actual project results , 
or the project will be eliminated from the data base. 

Suggestions 

The following next steps are suggested and include follow-on research: 

(Year 2013 to 2014) 

- Continue push to collect any remaining retrospective data from historical NASA technology projects. 

- Data Analysis to consolidate parameters selected for use. 

- Further TCASE Model development in the four TAs. 

- Investigate use of TechPort for data repository. 

- Begin analysis of Data for the remaining TAs. 

(Near Term) 

- Refine Data /Quality Requirements. 

- Continue TCASE Model use within NASA Cost Community for Beta Testing. 

- Develop CERs for use in Parametric Analysis. 

- Initiate Technology Estimating data input requirements /management requirements to TechPort. 
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(Long Term) 

- Include balance of Technology Areas (1, 2, 5, 6, 7, 9, 10, 11, 13, and 14). 

- Configure data pull from data sources such as TechPort. 

- Maintain and Update Data and TCASE Model. 

- Refine tool for NASA-wide deployment and use by Technologist, Management, and Projects. 

Conclusion 

The Technology Estimating process has been evaluated to determine its feasibility for providing a 
means to estimate technology. Based on the preliminary research, the process was determined to be a 
viable means to help determine the cost and schedule of technology for four of the 14 NASA Roadmap 
Technology Areas if the data was centralized, verifiable, and maintained. The research for Technology 
Estimating should continue to help refine and expedite the initial efforts initiated in this research. 

The following conclusions have been derived from the research: 

1. This Technology Estimating Research project addresses a gap in the NASA cost estimating 
capabilities for technologies in the range from TRL 2 to 6. 

2. The data collection work in the research has yielded valuable information from available sources, 
although only for a small subset of targeted parameters. These initial results have afforded the 
development of a useful analogy based estimation tool. However, retrospective data collection 
has been challenging and efforts are continuing, to enable rigorous analysis of parameters. 

3. A preliminary recommendation has been identified for data parameters to be used in the 
development of CERs to be applied to cost estimating using parametric analysis. 

4. The future goal, pending further analysis of all parameters, will lead to CER development and 
evolving capabilities from an analogy based cost estimating tool now to a parametric analysis tool 
for the initial four technology areas investigated, i.e. TA3, TA4, TA8 and TA12. 
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TECHNOLOGY ESTIMATING RESEARCH 

PROJECT 


Introduction and Definitions 

Date: March 18, 2013; Revised June 21, 2013 

The NASA HQ, Cost Analysis Division is performing research to improve the capabilities for estimating 
the cost and schedule of TRL 2 to 6 technology projects. For this research, the Research Team has helped 
to define the following 20 parameters. 


PARAMETERS 

1. Start Date and End Date 

The beginning and end date of the project. These dates will be used to determine the duration in years 
and months of the technology development project. 

2. Technology Readiness Level (TRL) 

A systematic measurement system that supports assessment of the maturity of a particular technology and 
the consistent comparison of maturity between different types of technologies. (1) The Technology 
Readiness Level (TRL) describes the stage of maturity in the development process from observation of 
basic principles through final product operation. The exit criteria for each level documents that principles, 
concepts, applications or performance have been satisfactorily demonstrated in the appropriate 
environment required for that level. TRL is measured on a scale of increasing maturity from 1 to 9 as 
shown in the following Table 1. (2) 


Technology Readiness Level 
Table 1 

TR 

L 

Definition 

Hardware Description 

Software Description 

Exit Criteria 

1 

Basic 
principles 
observed and 
reported 

Scientific knowledge 
generated underpinning 
hardware technology 
concepts/applications. 

Scientific knowledge 
generated underpinning basic 
properties of software 
architecture and 
mathematical formulation. 

Peer reviewed 
publication of 
research underlying 
the proposed 
concept/application. 

2 

Technology 
concept and/or 
application 
formulated 

Invention begins, 
practical application is 
identified but is 
speculative, no 
experimental proof or 
detailed analysis is 

Practical application is 
identified but is speculative, 
no experimental proof or 
detailed analysis is available 
to support the conjecture. 
Basic properties of 

Documented 
description of the 
application/concept 
that addresses 
feasibility and 
benefit. 
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available to support the 
conjecture. 

algorithms, representations 
and concepts defined. Basic 
principles coded. 
Experiments performed with 
synthetic data. 


3 

Analytical and 

experimental 

critical 

function 

and/or 

characteristic 

proof-of 

concept 

Analytical studies place 
the technology in an 
appropriate context and 
laboratory 
demonstrations, 
modeling and simulation 
validate analytical 
prediction. 

Development of limited 
functionality to validate 
critical properties and 
predictions using non- 
integrated software 
components. 

Documented 
analytical/ experiment 
al results validating 
predictions of key 
parameters. 

4 

Component 

and/or 

breadboard 

validation in 

laboratory 

environment 

A low fidelity 
system/component 
breadboard is built and 
operated to demonstrate 
basic functionality and 
critical test 
environments, and 
associated performance 
predictions are defined 
relative to the final 
operating environment. 

Key, functionally critical, 
software components are 
integrated, and functionally 
validated, to establish 
interoperability and begin 
architecture development. 
Relevant Environments 
defined and performance in 
this environment predicted. 

Documented test 
performance 
demonstrating 
agreement with 
analytical predictions. 
Documented 
definition of relevant 
environment. 

5 

Component 

and/or 

breadboard 

validation in 

relevant 

environment 

A medium fidelity 
system/ component 
brassboard is built and 
operated to demonstrate 
overall performance in a 
simulated operational 
environment with 
realistic support 
elements that 
demonstrates overall 
performance in critical 
areas. Performance 
predictions are made for 
subsequent development 
phases. 

End-to-end software 
elements implemented and 
interfaced with existing 
systems/ simulations 
conforming to target 
environment. End-to-end 
software system, tested in 
relevant environment, 
meeting predicted 
performance. Operational 
environment performance 
predicted. Prototype 
implementations developed. 

Documented test 
performance 
demonstrating 
agreement with 
analytical predictions. 
Documented 
definition of scaling 
requirements. 

6 

System/subsys 
tern model or 
prototype 
demonstration 
in a relevant 
environment 
(ground or 
space) 

A high fidelity 
system/ component 
prototype that 
adequately addresses all 
critical scaling issues is 
built and operated in a 
relevant environment to 
demonstrate operations 
under critical 

Prototype implementations 
of the software demonstrated 
on full-scale realistic 
problems. Partially integrate 
with existing 

hardware/software systems. 
Limited documentation 
available. Engineering 
feasibility fully 

Documented test 
performance 
demonstrating 
agreement with 
analytical predictions. 
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environmental 

conditions. 

demonstrated. 


7 

System 
prototype 
demonstration 
in a space 
environment 

A high fidelity 
engineering unit that 
adequately addresses all 
critical scaling issues is 
built and operated in a 
relevant environment to 
demonstrate 
performance in the 
actual operational 
environment and 
platform (ground, 
airborne, or space). 

Prototype software exists 
having all key functionality 
available for demonstration 
and test. Well integrated 
with operational 
hardware/software systems 
demonstrating operational 
feasibility. Most software 
bugs removed. Limited 
documentation available. 

Documented test 
performance 
demonstrating 
agreement with 
analytical predictions. 

8 

Actual system 
completed and 
“flight 
qualified” 
through test 
and 

demonstration 
(ground or 
space) 

The final product in its 
final configuration is 
successfully 
demonstrated through 
test and analysis for its 
intended operational 
environment and 
platform (ground, 
airborne, or space). 

All software has been 
thoroughly debugged and 
fully integrated with all 
operational hardware and 
software systems. All user 
documentation, training 
documentation, and 
maintenance documentation 
completed. All functionality 
successfully demonstrated in 
simulated operational 
scenarios. Verification and 
Validation (V&V) 
completed. 

Documented test 
performance 
verifying analytical 
predictions. 

9 

Actual system 

“flight 

proven” 

through 

successful 

mission 

operations 

The final product is 
successfully operated in 
an actual mission. 

All software has been 
thoroughly debugged and 
fully integrated with all 
operational 

hardware/software systems. 
All documentation has been 
completed. Sustaining 
software engineering support 
is in place. System has been 
successfully operated in the 
operational environment. 

Documented mission 
operational results. 


Note: The Technology Readiness Assessment Tool (TRAT, can be found at https://trl.msfc.nasa.gov/ ), 
developed by NASA Marshall Space Flight Center is available. If desired, the tool can assist in 
conducting an assessment of the beginning and ending TRL. 
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3. Total Cost 


Total dollars required to complete the technology development project. This data will be provided by 
year. Cost represents total cost of labor, materials, travel, testing and equipment, etc. and should also 
include (and separately identify) facilities and infrastructure capital investments made as part of the 
research project (if any). 

4. Estimate or Actuals 

A categorization of the data. 

• Actuals: Data collected from realized, historical technology development projects 

• Estimate: Data from studies, projections, and other sources that have not yet been realized 


5. Level of Effort 

A measure (in total work-years) of personnel workload required in terms of Full Time Equivalents (FTE) 
for civil servants, or Work Year Equivalent (WYE) for contractors. One work-year represents a single 
project staff member (either civil servant or contractor) working full time on the project for one year. For 
example, if a project has two individuals working full time during a three month period of performance, 
then the project has 0.5 work-years. 


6. Key Performance Parameter (KPP) 

Those capabilities or characteristics associated with the technology that is considered to be the primary 
objective of the technology development effort. Technology development programs seek to improve the 
current state-of-the-art, so the advancement of a program can be measured by the percentage 
improvement in one or more KPPs. (3) 

KPP information can include: 

• Description of KPP and units 

• Current value for state-of-the-art at the start of the project 

• Demonstrated achievable value at the end of the project 


For example, a jet engine propulsion technology development activity might have the goal to improve 
Specific Fuel Consumption (SFC). Thus SFC would be a KPP for this technology, the units would be 
defined (for example, g/[N*s]), along with values for state-of-the-art at the beginning of the project, and 
the demonstrated achievable improved value at completion of the project would be entered. The data 
would then be used to determine the percent improvement in the KPP as a result of the technology 
development project. 


Page 46 of 88 



7. Performing Organization 

A list of the names of the agencies, organizations, or companies conducting the technology development 
project. The number of organizations involved will be determined from this list and used as a data 
parameter. 

8. Team Experience 

Experience of the team conducting the research may be a big factor in how long the project takes and 
what it will cost. 

Four categories of research team experience are identified, as defined below. Research team experience is 
to be assessed at the beginning of the technology project. The categories and definitions are: 

1. The team has little or no experience conducting research. 

2. The team has experience conducting research, but little or no experience with this specific 
technology. 

3. The team has experience conducting research on this specific technology 

4. The team has experience conducting research on this specific technology and ALSO has 
incorporated members from outside the field(s) of research who can bring new ideas and sources of 
innovation to the team. 

9. Technology Area (TA) 

The fourteen areas of space technology research being pursued within NASA, plus the fundamental 
research area of aeronautics. Each technology area has an associated development roadmap. The list of 
space technology areas and their supporting roadmaps was developed by NASA and reviewed and 
validated by the National Research Council (NRC). This process began in 2010 and was completed in a 
final report in early 2012. (4) The space Technology Areas have 3 levels, (5) which will be available to the 
Technology Manager in pull-down menus for entry in the data sheet. Aeronautics is normally not 
addressed within the Space Technology Roadmap TA taxonomy, but is included here for information as 
an additional ("+1") area to fully represent the technology research pursued at NASA. 
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14+1 TECHNOLOGY AREAS 
Table 2 

TA# 

Description 

TA01 

Launch Propulsion Systems 

TA02 

In- Space Propulsion Technologies 

TA03 

Space Power and Energy Storage 

TA04 

Robotics, Telerobotics, Autonomous Systems 

TA05 

Communication and Navigation 

TA06 

Human Health, Life Support, Habitation Systems 

TA07 

Human Exploration Destination Systems 

TA08 

Science Instruments, Observatories, Sensor Systems 

TA09 

Entry, Descent, and Landing Systems 

TA10 

Nanotechnology 

TA11 

Modeling, Simulation, Information Tech 

TA12 

Materials, Structures, Mechanical Systems, Manufacturing 

TA13 

Ground and Launch Systems Processing 

TA14 

Thermal Management Systems 

(+)1 

Aeronautics 


10. System Hierarchy 

The scope of the technology project in terms of its application of a vehicle or system. The project can be 
classified in one of the following five tiers (6) as shown on Table 3. If the technology is a process or 
software, the tier indicates at which level of the system hierarchy the process or software is applied. The 
tiers are: 
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System Hierarchy 
Table 3 

No. 

Tier 

Definition 

Example 

1 

System 

An integrated set of constituent 
elements that are combined in an 
operational or support environment to 
accomplish a defined objective 

A spacecraft or launch 
vehicle stage 

2 

Subsystem 

A portion of a system 

A satellite’s propulsion 
system or launch vehicle’s 
propulsion system 

3 

Assembly 

A set of components (as a unit) before 
they are installed to make a final 
product 

A satellite’s thruster or 
launch vehicle’s engine 
turbo-machinery 

4 

Component / Part 

A portion of an assembly 

A satellite’s propellant valve 
or a launch vehicle’s engine 
injector 

5 

Hardware / Material 

An item or substance used to form a 
component 

Alloy, polymer, screws, 
bolts, pipes, semiconductor 
chips 


11. Hardware vs. Software 

A top-level assessment of whether the end application of the technology is: 

• Hardware: The physical items necessary for conducting an activity, as distinguished from the 
theory and design that make the activity possible. Examples include mechanical equipment, tools, 
implements, instruments, devices, sets, fittings, trimmings, and assembled collections of those 
items. 

• Software: Computer programs, procedures, scripts, rules, and associated documentation and data 
pertaining to the development and operation of a computer system. Software includes programs 
and data. This also includes COTS, GOTS, MOTS, reused software, auto generated code, 
embedded software, firmware, and open source software components. (7) 

12. Research and Development Degree of Difficulty (R&D 3 ) 

A top-down assessment of the anticipated difficulty likely to be encountered over the course of a 
technology maturation project (e.g. from the beginning TRL to end TRL). (9) R&D 3 is a qualitative 
evaluation of the probability of success of the development project. R&D 3 is assessed on a scale of 
increasing difficulty from 1 to 5 as shown on Table 4: 
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Research and Development Degree of Difficulty 
Table 4 

R&D 3 

Definition 

Probability 
of Success 

1 

A very low degree of difficulty is anticipated in achieving research and 
development objectives for this technology. 

> 95%-99% 

2 

A moderate degree of difficulty should be anticipated in achieving R&D 
objectives for this technology. 

> 90% 

3 

A high degree of difficulty anticipated in achieving R&D objectives for this 
technology. 

> 80% 

4 

A very high degree of difficulty anticipated in achieving R&D objectives for 
this technology. 

~ 50-60% 

5 

The degree of difficulty anticipated in achieving R&D objectives for this 
technology is so high that a fundamental breakthrough is required. 

< 30% 


13. Advancement Degree of Difficulty (AD 2 ) 

A bottoms-up assessment of the anticipated difficulty over the course of a technology maturation project 
(e.g. from the beginning TRL to end TRL). (9) AD 2 is determined through consideration of cost, schedule, 
and risk across several dimensions, including: 

• Design and Analysis 

• Manufacturing 

• Software and Development 

• Test and Evaluation 

• Operations 

• AD 2 is intended to be determined using the AD 2 calculator found at https://trl.msfc.nasa.gov/, or 
with a resulting value on a scale of increasing difficulty from 1 to 9 as shown on Table 5: 


Advancement Degree of Difficulty 
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Table 5 

AD 2 

Definition 

Risk 

Category 

Success 

Chance 

1 

Exists with no or only minor modifications being 
required. A single development approach is adequate. 

0% 


Guaranteed 

Success 

2 

Exists but requires major modifications. A single 
development approach is adequate. 

10% 



3 

Requires new development well within the experience 
base. A single development approach is adequate. 

20% 



4 

Requires new development but similarity to existing 
experience is sufficient to warrant comparison across 
the board. A single development approach can be taken 
with a high degree of confidence for success. 

30% 

Well 

Understood 

(Variation) 

Almost Certain 
Success 

5 

Requires new development but similarity to existing 
experience is sufficient to warrant comparison in all 
critical areas. Dual development approaches should be 
pursued to provide a high degree of confidence for 
success. 

40% 

Known 

Unknowns 

Probably Will 
Succeed 

6 

Requires new development but similarity to existing 
experience is sufficient to warrant comparison on only a 
subset of critical areas. Dual development approaches 
should be pursued in order to achieve a moderate degree 
of confidence for success. (Desired performance can be 
achieved in subsequent block upgrades with high 
confidence. 

50% 



7 

Requires new development but similarity to existing 
experience is sufficient to warrant comparison in only a 
subset of critical areas. Multiple development routes 
must be pursued. 

70% 



8 

Requires new development where similarity to existing 
experience base can be defined only in the broadest 
sense. Multiple development routes must be pursued. 

80% 

Unknown 

Unknowns 

High 

Likelihood of 
Failure (High 
Reward) 

9 

Requires new development outside of any existing 
experience base. No viable approaches exist that can be 
pursued with any degree of confidence. Basic research 
in key areas needed before feasible approaches can be 
defined. 

100% 

Chaos 

Almost Certain 
Failure (Very 
High Reward) 


Note: If needed, the AD 2 Calculator is a tool that can assist the Technology Manager in conducting an 
objective assessment. The AD 2 calculator is co-located with TRAT. 
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14. Technology Push or Pull 

• Technology Push: Technologies that seek to meet long term goals by taking advantage of 
emerging or radically different ideas or approaches. Push technologies inspire new and different 
Missions and Mission Architectures to accomplish strategic long-term goals. (10) 

• Technology Pull: Technologies that support or enable specific functions, capabilities or 
performance levels, for a defined mission, that is not achievable by proven, existent approaches. 
Pull technologies are motivated by the specific needs for missions, where the missions are 
developed by enterprise organizations to satisfy long-term strategic goals. (10) 

15. Evolutionary or Revolutionary 

• Evolutionary: An incremental or stepwise improvement from a more mature, less advanced 
technology. 

• Revolutionary: A fundamentally new concept without a direct analogy, or a disruptive new 
innovation that is a significant advancement from an existing analogy. 


16. Funding Sources 

A list of the names of the organization(s), program(s), or contract(s) providing funding to the technology 
development project. 

17. Facilities and Infrastructure 

A list of the facilities and major infrastructure items required in the support of the technology 
development project. These could include: commercial or government laboratories, buildings, test stands, 
wind tunnels, etc. The number of facilities and infrastructure items will be determined from this list and 
used as a data parameter. 


18. Budget Constraints and Disruptions 

A list of the external constraints and disruptions to the resources required for the technology development 
project that may have negatively influenced the required project cost or schedule. For disruptions that 
influenced schedule, such as a break in funding between phases, the duration of these disruptions should 
be included. The number of total budget constraints and disruptions will be determined from this list and 
used as a data parameter. 


19. Unplanned Events 

A list of events outside of the control of the technology development project that had a negative effect on 
the required project cost or schedule. Examples include natural disasters, realignment of government or 
organization priorities, retirement or reassignment of key personnel, strikes, embargos, or testing 
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incidents or accidents, etc. The number of total unplanned events will be determined from this list and 
used as a data parameter. 


20. Defining System Characteristics 

Technical parameters or characteristics that broadly define the scope of the technology project, but are not 
objectives of the technology development effort. For example, consider an energy storage technology 
development project with the objective of increasing Li-ion battery subsystem reliability to a goal value 
of 99.9%. In this case, parameters such as the energy density and the efficiency of the test battery are not 
direct objectives for improvement, but they may still be listed as Defining System Characteristics. (The 
battery subsystem reliability, on the other hand, should be listed as a Key Performance Parameter.) 
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Technology Estimating - Data Source Survey 

NASA HQ Office of Evaluation, Cost Analysis Division, Technology Cost and Schedule Estimating Research Project 
(Version: Rev A, Issue Date 11-15-2012) 


Data Source Name 


Name 

Organization 
Email Address 
Work Phone 


Description 
Tech Area(s) 


Respondent 


Data Source POC 


Type of Data (Respond as applicable): 

Historical Actual \^\ Historical Estimates \^\ 


Number of Projects | 


Technology Project Size, Each (Mark all that apply) 

Less than $1M j j 


$1M to $250M 


n 


Active Projects 
Greater than $250M | j 


□ 


Tracked Technology Parameters (Mark all that apply): 


Cost 


TRL 


Technology Area 


Facilities & Infrastructure Reqr'mnts 


Complexity 


FTEs 


Schedule 


R&D 3 


Tech. Proj. Tier /Scope 


Funding Source 


Push vs. Pull 


Unplanned Events 


Programmatic Risk 


A&D 2 


A or TRL Progression 


Agencies/ORG/Companies Involved 


Funding Source 


Actual vs. Estimates 



Perfomance Characteristics / KPPs, status and goal, perTA 
Capture progression/Change of Parameters overtime 
Projected Metric & Theoretical Limit 


Evolutionary Vs. Revolutionary 
Budget Constraints & Disruptions 
SOP & SOA at start of Tech. Development 


KPPs (list or describe) 
Other (describe) 


Platform (e.g. MS Excel) 
Date Created 
Update Frequency 
Last Update 


Data Sens itivity ( Mark all that apply) 

SBU I | 

Other (describe) 


Classified or Restricted 


□ 


Contractor Proprietary 


□ 


NOTES: 

1. You may print this sheet, hand write the answers, and then electronically scan it for email. 

2. Please complete form and email to Stuart.Cole@nasa.gov. Call at 757-951-7403 if there are any questions. 

3. Multiple sheets are acceptable. 

4. Complete the information to the best of your knowledge, cite sources for future reference. 

5. The purpose of the "Survey" is to obtain the best characterization of the data that is availiable. 
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Tier 

Data Source Name 

Full Name 


Primary Sources 

1 

i 

NTID 

NASA Tech Inventory Database 

2 

i 

Cx ETDP Portfolio 


3 

i 

ARC/Mathias Lunar Surface Data 


4 

i 

External Gov Tech Maturation Data 

Tauri Group research into External Gov Technology Tech 
Maturation data 

5 

i 

Sefcik Analysis 

GRC Analysis of NTID 

6 

i 

ESMD R&TTech Data Sheets (2006) 


7 

i 

RLVTech DB (1993) 

NASA RLV Technology Database 1993 

8 

i 

HEOMD TechDev Team 


9 

i 

ATLAS/TITAN 

Advanced Technology Lifecycle Analysis System 

10 

i 

High Speed Research Program (80s-90s) 

Wilhite led High Speed Research Program 

11 


National Aerospace Plane 


12 


NASA SBIR/STTR 


13 

i 

HEOMD NSPIRES 


14 

i 

SMD NSPIRES 


15 

i 

TBCC Tech Assessment (2010) 

Turbine Based Combined Cycle Tech Assessment Technology 
Data Sheets 2010 

16 

i 

HLS ACENT 

DARPA/NASA Horizontal Launch Study ACENT Follow-On 

17 

i 

Gryphon Phase 2A Technologies 

Gryphon Study Phase IIA High Speed Flight Experiments Concept 
Study Technologies 

18 

i 

PIT 

Program Integration Tool 

19 

i 

GCD Project Snapshot 


20 

i 

TechPort 


21 

i 

START 

Strategic Assessment of Risk and Technology 

22 


DLR Data set 


23 


IDA Dataset 


24 


DoD Technology Mall 



Tools and Templates 

25 

1 

TCE 

Technology Cost Estimator 

26 

1 

AFRL Technology Calculator 


27 

1 

Bilbro Technology Calculator 

Technology Assessment Calculator 

29 


Bilbro AD 2 Calculator 


30 

1 

DHS SS&T Readiness Level Calculator 

31 


NAVAIR spreadsheet 


32 


NICM Template 


33 


DARPA Business Acumen Assessment 

34 


RI3 Guidebook from Airforce 


35 

II 

AFRL Projects 


36 

II 

DARPA Projects 


37 

II 

DOD/Army SBIR/STTR 
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38 

II 

DO D/M DA SBIR/STTR 


39 

II 

DOD/Navy SBIR/STTR 


40 

II 

DOD/USAF SBIR/STTR 


41 

II 

DOE Projects 


42 

II 

NASA NIAC 1 and II 


43 

II 

NASA OCT (Early Stage, Game Changing, Cross Cutting) 

44 

II 

NASA SBIR/STTR 


45 

II 

NRL Projects 


46 

II 

NSF Grants 


47 

II 

NSF SBIR/STTR 


48 

II 

SBIR "Enhancement Awards" 


49 

1 

CADRe 

Cost Analysis Data Requirement 

50 

III 

Advanced Product Design Team (Team X) Cost 
Model 

Advanced Product Design Team (Team X) Cost Model 

51 

III 

ALS/ADP Cost Model 

ALS/ADP Cost Model 

52 

III 

A-PICOMO 

Aerospace Picosatellites Cost Model 

53 

III 

BOE Tool 

Basis of Estimate Tool 

54 

III 

CEV TRL Summary/Details 


55 

III 

ChiCoMo 

Chicago Cost Model 

56 

III 

ICSAT 

Integrated Cost and Schedule Analysis Tool 

57 

III 

MATCH Database 

Mapping Applicable Technology To Exploration Challenges 

58 

III 

MICM 

Multivariable Instrument Cost. Model 

59 

III 

NAFCOM 2011 

NASA / Air Force Cost Model 

60 

III 

NAFCOM/REDSTAR 


61 

III 

NICM 

NASA Instrument Cost Model 

62 

III 

Parametric Cost Modeling for Space Telescopes 

Parametric Cost Modeling for Space Telescopes 

63 

III 

PMCM 

Planetary Mission Cost Model 

64 

III 

PRICE TruePlanning 

PRICE TruePlanning 

65 

III 

PRICE-H 

PRICE for Hardware 

66 

III 

PSCM 

Passive Sensor Cost Model 

67 

III 

QuickCost 5.0 

QuickCost 5.0 

68 

III 

SEER-EOS (formerly SEER-Spyglass) 

SEER for Electro Optical Sensors 

69 

III 

SEER-H 

SEER for Hardware, Electronics, & Systems 

70 

III 

SEER-IC 

SEER for Integrated Circuits 

71 

III 

SEER-MFG 

SEER for Manufacturing 

72 

III 

SEER-SEM 

SEER for Software 

73 

III 

Sensat 

Sensor Satellite Expert System 

74 

III 

SICM 

Scientific Instrument Cost Model 

75 

III 

SSCM10 

Small Satellite Cost Model 2010 

76 

III 

VIEW 

VIEW 
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Data Collection Sheets 


Data Collection Sheets are available from the Cost Analysis Division. 
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APPENDIX E 


Technology Estimating Data Quality and Screening Methodology 
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To: Doug Comstock, CAD 

From: Research Team 

Re: Technology Estimating Data Quality and Screening Methodology 

Date: January 30, 2013; Revised June 10, 2013 


The purpose of this paper is to help elicit discussion leading to formulation of a method for screening data 
for this research. The purpose for screening data is to obtain quality Technology data in the Technology 
Estimating research project. The screening methodology is also intended to provide guidance for selection 
of data to be used in the effort, and which data should be applied, or discarded. 

Currently, based on the Phase I meeting, technology project data is to be obtained for low TRL (2 to 6) 
technology projects and for NASA Technology Roadmap areas defined by TA 03, 04, 08, and 12. There 
are 20 parameters to be investigated for each of the technologies and to help determine Cost Estimating 
Relationships. The effort to determine CERs from these 20 parameters generally requires generation of 
the information from persons intimately knowledgeable of the project. 

Generally, the research effort is to help define which of the 20 parameters are beneficial to use for 
development of CERs. In addition, development of a model requires use of data that has been carefully 
reviewed by the Research Team for any inconsistencies or are poorly developed which might lead to 
misleading conclusions as to which of the parameters are of benefit towards the preparation of a point 
cost and schedule estimate with a reported accuracy range. 

Initially, for analysis of the 20 parameters, it is perceived that the data used in the model will help lead 
development and reporting of the cost and schedule accuracy range based on the quality data used. It is 
proposed that this accuracy range should perceptively change with the quality of data used. 

Upon completion of the data collection process with the Technology Managers expected in the 1 st 
Quarter 2013, it should become apparent which of the 20 parameters will be eliminated. The elimination 
of any parameter for use in the model will need to be documented. The determination should include a 
narrative for why the parameter was eliminated, discuss any misconceptions, the “feel” as it would apply, 
or list other shortcomings/notes/comments in definition or interpretation by any of the Technology 
Research Team or research participants. 
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SCREENING CRITERIA 


1. Technology Projects would be categorized into high, medium, low, or no/questionable value. 

2. High value: Technology projects could be identified as high value would be completed work 
showing TRL progression from TRL 2 to TRL 6 and/or beyond. These projects most likely would 
be represented by: 

• SBIR/STTR projects Phase III 

• ESTO, 

• EDTP, 

• Game Changing Development Program Office projects, 

Data would be used to satisfy or resolve of the 20 parameters would be generated by Technology 
Managers and/or those persons with direct project knowledge. 

3. Medium value: These Technology Project data includes some or satisfy part of the 20 parameters 
without confirmation by a Technology Manager or project knowledgeable person. These data 
would be obtained from currently published or documented resources and include description of 
the technology, TA identification, beginning and end costs, dates, and TRL. The medium value 
data would include a citation for the source from where it was derived, i.e. traceable. 

4. Low value: The Low value Technology Projects data include any one of the following: 

• Un-traceable with no citation for source of data, 

• FY dollars are not identified, 

• Project data does not indicate the total project time, 

• Project is active and a Technology Manager can or cannot be found to verify the data, 

• Technology Manager can verify project future end point costs, TRL progression, 

• Only one cost is provided. 

5. No/Questionable value: These criteria include technology data where anyone of the following 
applies: 

• Traceability does not exist, 

• The project is active, and no Technology Manager cannot be found to verify the data, 

• The origin of the data is in question, 

• Beginning or end cost data has not been provided, 

• No TRL data exists, 

• Project descriptions are not well defined, 

• No TA data, or 

• Basic technology project data is in question. 

The No/Questionable project data should not be used in the research. 
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APPENDIX F 


Parameter Evaluation Narrative 
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Schedule/Cost driver: a parameter that tends to increase or decrease schedule/cost or is highly correlated with 
schedule/cost 

To qualify as a cost driver, a parameter must be a significant predictor of schedule/cost at least in the statistical sense 


Project 

Characterization 

Metric 

Attribute 

Rank Ease of 
Administration 
Low, Med. High 

Schedule 

Driver 

Cost 

Driver 

Recommended for 
Use in Future Data 
Collection 

Technology Area 
(TA) 

Qualitative 

High 

Yes 

Yes 

Yes 


1 . Technology areas in secondary and tertiary breakdown were sometimes difficult to 
apply in all cases. 

2. This parameter only applied to recent projects as the 14 TA Roadmap was 
promulgated within the last two years. 

3. TA significantly helps to refine the Technology however branching TAs can or may 
be questioned. 

4. Different individual TAs are expected to have generally different schedule and 
costs. 

5. This parameter is included in TechPort. 



Schedule/Cost driver: a parameter that tends to increase or decrease schedule/cost or is highly correlated with 
schedule/cost 

To qualify as a cost driver, a parameter must be a significant predictor of schedule/cost at least in the statistical sense 


Project 

Characterization 

Metric 

Attribute 

Rank Ease of 
Administration 
Low, Med. High 

Schedule 

Driver 

Cost 

Driver 

Recommended for 
Use in Future Data 
Collection 

System Hierarchy 

Qualitative 

Med. 

Yes 

Yes 

Yes 


1 . This parameter was not applied easily by Technology Managers. 

EXAMPLE: If a satellite (mission i.e. “platform" or "system of systems”) had multiple 
sub-systems comprising the total science package, it could not readily be determined 
that a technology was itself a complete system or was it to be included in the larger 
science system or platform. 

2. Would benefit from a standard reference to help clarify appropriate level in the 
hierarchy. 
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Schedule/Cost driver: a parameter that tends to increase or decrease schedule/ cost or is highly correlated with 
schedule/cost 


To qualify as a cost driver, a parameter must be a significant predictor of schedule/cost at least in the statistical sense 


Project 

Characterization 

Metric 

Attribute 

Rank Ease of 
Administration 
Low, Med. High 

Schedule 

Driver 

Cost 

Driver 

Recommended for 
Use in Future Data 
Collection 

Hardware vs. 
Software 

Qualitative 

Low 

Yes 

Yes 

Yes 


1 . Recommend that "Hardware” should be dropped as it does not provide 
significant correlation to amount or extent of software in all technologies. 

2. Suggested Typical software cost drivers include: Source lines of 
code, Function point count, Object class count, Number of screens, 
Language, Application type, Complexity (various measures), Build 
environment, Target environment, Reliability, Maintainability, Degree of 
reuse or COTS, Number FTEs, Team capability. 

3. Software development can be expected to be significant in the range of 
TRL 5 to 6. 

4. Technology papers indicate sometimes one or two persons exclusively 
dedicated to software. And extent of software can be major influence to 
cost and success of the technology. 

5. Data may exist as part of NPR 7150.2A software management. 



Schedule/Cost driver: a parameter that tends to increase or decrease schedule/cost or is highly correlated with 
schedule/cost 

To qualify as a cost driver^ a parameter must be a significant predictor of schedule/cost at least in the statistical sense 


Project 

Characterization 

Metric 

Attribute 

Rank Ease of 
Administration 
Low, Med. High 

Schedule 

Driver 

Cost 

Driver 

Recommended for 
Use in Future Data 
Collection 

Push vs. Pull 

Qualitative 

High 

No 

No 

TBD 


1 . The parameter goes to intent of the Technology initiation and that 
determination can be quite difficult to determine and is possibly only 
known by a few, may never be documented, or is obscure. 

2. If there is push then only a few discrete projects may be classified and 
these may be an exception, 

3. If there is a push, then this parameter might be added later. 

4. Currently, this parameter is recommended to be evaluated further. 
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Schedu I e/Cost driver: a parameter that tends to increase or decrease schedule/cost or is highly correlated with 
schedule/cost 


To qualify as a cost driver, a parameter must be a significant predictor of scheduleJcost at least in the statistical sense 


Project 

Characterization 

Metric 

Attribute 

Rank Ease of 
Administration 
Low, Med. High 

Schedule 

Driver 

Cost 

Driver 

Recommended for 
Use in Future Data 
Collection 

Evolutionary vs. 
Revolution 

Qualitative 

Low 

No 

No 

No 


1 , The parameter goes to intent of the Technology initiation and that 
determination can be quite difficult to determine and is possibly only 
known by a few, may never be documented, or is obscure. 

2, If there is revolutionary technologies then only a few discrete projects 
may be classified and these may be an exception. 

3, Currently, this parameter is recommended to be eliminated. 



Schedule/Cost driver: a parameter that tends to increase or decrease schedule/cost or is highly correlated with 
schedule/cost 

To qualify as a cost driver, a parameter must be a significant predictor of schedule/cost at least in the statistical sense 


Project 

Characterization 

Metric 

Attribute 

Rank Ease of 
Administration 
Low, Med, High 

Schedule 

Driver 

Cost 

Driver 

Recommended for 
Use in Future Data 
Collection 

Defining System 

Quantifiable 

Med. 

Yes 

Yes 

Yes 


Characteristics 


1 . Provides the detailed characterization of TAs. 

2. Helps to characterize between technologies in same TA and for branching 
technologies from a single TA. 

3. This parameter includes mass, volume, power, etc. 

4. With future data, the characterizations could reveal for some TAs i.e. 08 
Instrumentation, that they follow Moore’s Law where "number of transistors on 
integrated circuits doubles approximately every two years” or possess smaller 
volumes, and less mass due to increased computing capability. 
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Schedule/Cost driver: a parameter that tends to increase or decrease schedule/cost or is highly correlated with 
schedule/cost 

To qualify as a cost driver, a parameter must be a significant predictor of schedule/cost at least in the statistical sense 


Project Results 

Metric 

Attribute 

Rank Ease of 
Administration 
Low, Med. High 

Schedule 

Driver 

Cost Driver 

Recommended for 
Use in Future Data 
Collection 

Key Performance 
Parameter (KPP) 

Quantifiable 

High 

Yes 

Yes 

Yes 


1 . This parameter includes any and all project objectives. 

2. The KPPs provide significant and relevant data about the project and its 
intended functional objectives. 

3. This parameter provides the description in sufficient detail to compare it against 
similar technology projects. 

4. This parameter also provides technical details such that branching applications 
may be understood. 

5. When adequate data is provided, technologies with similar objectives may be 
compared by their respective schedule and cost. 



Schedule/Cost driver: a parameter that tends to increase or decrease schedule/cost or is highly correlated with 
schedule/cost 

To qualify as a cost driver, a parameter must be a significant predictor of schedule/cost at least in the statistical sense 


Project Results 

Metric 

Attribute 

Rank Ease of 
Administration 
Low, Med. High 

Schedule 

Driver 

Cost Driver 

Recommended for 
Use in Future Data 
Collection 

Level of Effort 

Quantifiable 

High 

No 

Yes 

Yes 


1 . This parameter includes manpower estimate for NASA but does not capture the 
cost of contractors, universities, students, etc. 

2. A consideration should be made to add the capture the contracting costs in 
WYEs. 
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Schedule/Cost driver: a parameter that tends to increase or decrease schedule/cost or is highly correlated with 
schedule/cost 


To qualify as a cost driver, a parameter must be a significant predictor of schedule/cost at least in the statistical sense 


Project Results 

Metric 

Attribute 

Rank Ease of 
Administration 
Low, Med. High 

Schedule 

Driver 

Cost Driver 

Recommended for 
Use in Future Data 
Collection 

Cost 

Quantifiable 

High 

Yes 

Yes 

Yes 


1 . This parameter includes the total cost (actual). 

2. It is suggested that the cost per year be obtained and identified with the 
respective year. 

3. The cost that should be identified is the cost in the management information 



Schedule/Cost driver: a parameter that tends to increase or decrease schedule/cost or is highly correlated with 
schedule/cost 

To qualify as a cost driver, a parameter must be a significant predictor of schedule/cost at least in the statistical sense 


Project Results 

Metric 

Attribute 

Rank Ease of 
Administration 
Low, Med. High 

Schedule 

Driver 

Cost Driver 

Recommended for 
Use in Future Data 
Collection 

Schedule 

Quantifiable 

High 

Yes 

Yes 

Yes 


1 . This parameter includes the total duration (actual). 

2. It is suggested that the total duration be identified by beginning and end dates. 

3. The duration that should be identified is the time in the management information 
system, i.e. SAP or by official execution documents, i.e. traceable and input to 
the TechPort. 
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Schedule/Cost driver: a parameter that tends to increase or decrease schedule/cost or is highly correlated with 
schedule/cost 

To qualify as a cost driver, a parameter must be a significant predictor of schedule/cost at least in the statistical sense 


Development 

Metrics 

Metric 

Attribute 

Rank Ease of 
Administration 
Low, Med. High 

Schedule 

Driver 

Cost Driver 

Recommended for 
Use in Future Data 
Collection 

Technology 
Readiness Level 
(TRL) 

Qualitative 

High 

Yes 

Yes 

Yes 


1 . This parameter includes the Technology Readiness Level as it is submitted by 
the Technology Manager, PI, PM or Technical Paper. 

2. It is suggested that the TRL for a technology be assessed by a unbiased 
referee. 



Schedule/Cost driver: a parameter that tends to increase or decrease schedule/cost or is highly correlated with 
schedule/cost 

To qualify as a cost driver, a parameter must be a significant predictor of schedule/cost at least in the statistical sense 


Development 

Metrics 

Metric 

Attribute 

Rank Ease of 
Administration 
Low, Med. High 

Schedule 

Driver 

Cost Driver 

Recommended for 
Use in Future Data 
Collection 

Research and 
Development 
Degree of 
Difficulty (R&D 3 ) 

Qualitative 

Low 

Yes 

Yes 

TBD 


1 . This parameter was difficult to administer. 

2. The use of this parameter in this research was impeded by subjectivity. 
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Schedule/Cost driver: a parameter that tends to increase or decrease schedule/cost or is highly correlated with 
schedule/cost 

To qualify as a cost driver, a parameter must be a significant predictor of schedule/cost at least in the statistical sense 


Development 

Metrics 

Metric 

Attribute 

Rank Ease of 
Administration 
Low, Med. High 

Schedule 

Driver 

Cost Driver 

Recommended for 
Use in Future Data 
Collection 

Advancement 
Degree of 
Difficulty (AD 2 ) 

Qualitative 

Low 

Yes 

Yes 

TBD 


1 . This parameter was difficult to administer. 

2. The use of this parameter in this research was impeded by its subjectivity. 



Schedule/Cost driver: a parameter that tends to increase or decrease schedule/cost or is highly correlated with 
schedule/cost 

To qualify as a cost driver, a parameter must be a significant predictor of schedule/cost at least in the statistical sense 


Project 

Execution 

Metric 

Attribute 

Rank Ease of 
Administration 
Low, Med. High 

Schedule 

Driver 

Cost Driver 

Recommended for 
Use in Future Data 
Collection 

Funding 

Source(s) 

Qualitative 

High 

Yes 

Yes 

Yes 


1 . Comments from Technology Managers revealed that certain funding sources such 
IRAD programs do not provide enough time in the funding program to fully 
accomplish the Technology project efforts. 

2. NASA undergoes a mix of funding programs, i.e. ETDP to GCPDO, etc. and the 
funding sources help to identify tracking of the technology and for branching of the 
technology. 

3. Tracking technologies from lower TRL to higher TRL, or from incubation to 
mission level or cessation, is an imperative for NASA technology infusion. 
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Schedule/Cost driver: a parameter that tends to increase or decrease schedule/cost or is highly correlated with 
schedule/cost 

To qualify as a cost driver, a parameter must be a significant predictor of schedule/cost at least in the statistical sense 


Project 

Execution 

Metric 

Attribute 

Rank Ease of 
Administration 
Low, Med. High 

Schedule 

Driver 

Cost Driver 

Recommended for 
Use in Future Data 
Collection 

Organization(s) 

Involved 

Qualitative 

High 

No 

No 

Yes 


1 . Identification of organizations involved helps to identify tracking of the technology 
and for branching of the technology. 

2. The identification of organizations and points of contact is an imperative for 
tracking technology. 

3. There could be a schedule or driver if a certain organization is identified with the 
Technology. 

4. The parameter is included in TechPort. 



Schedule/Cost driver a parameter that tends to increase or decrease schedule/cost or is highly correlated with 
schedule/cost 

To qualify as a cost driver, a parameter must be a significant predictor of schedule/cost at least In the statistical sense 


Project 

Execution 

Metric 

Attribute 

Rank Ease of 
Administration 
Low, Med. High 

Schedule 

Driver 

Cost Driver 

Recommended for 
Use in Future Data 
Collection 

Facilities and 
Infrastructure 

Quantifiable 

High 

Yes 

Yes 

Yes 


1 . Identification of the facilities or infrastructure goes directly to the schedule and 
cost driver of a technology. 

2. The use of aircraft for testing was expected to be a large expenditure however it 
was not documented. 

3. Some laboratories were noted in the literature to be specialized and there should 
have been a cost associated with these labs. 

4. Certain University test facilities by were noted in discussion with Technology 
Managers to severely impact the delivery date of a technology project. These 
types of changed conditions would be noted in project administration records and 
should be identified. 
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Schedule/Cost driver: a parameter that tends to increase or decrease schedulefcost or is highly correlated with 
schedule/cost 


To qualify as a cost driver, a parameter must be a significant predictor of schedulelcost at least in the statistical sense 


Project 

Execution 

Metric 

Attribute 

Rank Ease of 
Administration 
Low* Med- High 

Schedule 

Driver 

Cost Driver 

Recommended for 
Use in Future Data 
Collection 

Team Experience 

Quantifiable 

High 

Yes 

Yes 

Yes 


1 . Identification of the team structure goes directly to the schedule and cost driver of 
a technology. 

2. The team structure was not well documented in any of the data sources. 

3. The team makeup would be noted in project administration records and should be 
identified. 



Schedule/Cost driver: a parameter that tends to increase or decrease schedule/cost or is highly correlated with 
schedule/cost 

To qualify as a cost driver, a parameter must be a significant predictor of schedule/cost at least in the statistical sense 


Programmatic 

Factors 


Metric 

Rank Ease of 

Schedule 

Cost Driver 

Recommended for 

Attribute 

Administration 

Driver 


Use in Future Data 


Low, Med. High 



Collection 


Estimates vs. Quantifiable High Yes Yes 

Actuals 


No 


1 . This parameter is recommend for elimination as all cost data collection in the 
future must be based on actuals and traceable data. 

2. Estimates should be replaced with actual data acquired from the project 
management information system (SAP) with actual costs broken down. 

3. A qualification to this cost data is that funding dollars should be broken down 
by yearly expenditures if possible and the funding year cost date be identified 
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Schedule/Cost driver: a parameter that tends to increase or decrease schedule/cost or is highly correlated with 
schedule/cost 

To qualify as a cost driver, a parameter must be a significant predictor of schedule/cost at least in the statistical sense 


Programmatic 

Factors 

Metric 

Attribute 

Rank Ease of 
Administration 
Low, Med. High 

Schedule 

Driver 

Cost Driver 

Recommended for 
Use in Future Data 
Collection 

Budget 

Constraints and 

Qualitative 

Low 

Yes 

Yes 

No 


Disruptions 


1 . This parameter is recommend for elimination as a single parameter and 
included with Unplanned Events. 

2. Project managers do not conclude that budget constraints or disruptions in 
budgeting will impact the project. 

3. Budget constraints or disruptions are difficult to use as a qualification. 

4. This parameter would or should be included in the “unplanned event” 



Schedule/Cost driver: a parameter that tends to increase or decrease schedule/cost or is highly correlated with 
schedule; co st 

To qualify as a cost driver, a parameter must be a significant predictor of schedule/cost at least in the statistical sense 


Programmatic 

Factors 

Metric 

Attribute 

Rank Ease of 
Administration 
Low, Med. High 

Schedule 

Driver 

Cost Driver 

Recommended for 
Use in Future Data 
Collection 

Unplanned 

Events 

Qualitative 

High 

Yes 

Yes 

Yes 


1 . This parameter includes any and all events significant to the capture of any or 
all impacts to the project such as change orders, requirements change, or 
unforeseen conditions. 

2. It is suggested that this parameter be changed to read “significant events”. 
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APPENDIX G 


Technology Cost and Schedule Estimating (TCASE) Tool 


The tool is available from the Cost Analysis Division. 
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APPENDIX H 


User Manual 

Technology Cost and Schedule Estimating (TCASE) Tool 
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User Manual 


TCASE 

Technology Cost and Schedule Estimating 

Beta Version 


Document Revision 0.3 
30 August 2013 


Developed by: 

SpaceWorks Software 

A Division of SpaceWorks Enterprises, Inc. (SEI) 


Funded by: 

NASA Cost Analysis Division (CAD) and Game 
Changing Development (GCD) Program Office 
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User Manual 


Contents 

Introduction 3 

List of TCASE Worksheets 3 

Front End Worksheet 4 

Search Criteria 4 

Statistical Results 4 

Top Analogies 5 

Analogy Manager Worksheet 6 

Data Viewer Worksheet 6 

Data Importer Worksheet 7 

Data Exporter Worksheet 8 

Database Worksheet 9 

Similarity Matrices Worksheet 10 

Frequently Asked Questions 10 

Acknowledgements 11 

Appendix A - Running a Simple Search 12 

Appendix B - Sources of Data Used in TCASE 13 


Technology Cost and Schedule Estimation (TCASE) Tool 


^SpaceWorks 

' Software 


User Manual | Rev0.3 (2013-08-30) 






rCASE: Technology Cost and Schedule Estimation 
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Introduction 


NASA and SpaceWorks have identified a need within the cost and schedule estimating community to rapidly assess the 
cost and schedule associated with maturing technologies along the Technology Readiness Level (TRL) scale. In response 
to this need, SpaceWorks has developed a Microsoft Excel® based software application called the Technology Cost and 
Schedule Estimation (TCASE) tool. 

TCASE generates anticipated ranges of cost and schedule duration for a technology development project by drawing 
analogies to historical and current project. Data for historical and current projects is stored in an accompanying 
database. The tool is specifically designed to examine technologies in the range of TRL 2 through TRL 6. 

At the end of this first year of research and development (September 2013), a prototype version of TCASE will be 
delivered to our research sponsors for testing and evaluation. The long-term goal of this research project is position 
TCASE as an everyday tool used by the NASA cost estimating and technology assessment communities. 


The TCASE Excel® workbook contains several worksheets that the user should become familiar with. The content and 
function of each is explained here. 


List of TCASE Worksheets 



Front End 


The primary point of interaction with the TCASE tool. Enter search criteria, perform 
project analogy searches, view statistical results, and examine the top analogies. 


Analogy Manager 


View all analogy results returned by a search. Turn individual analogies ON or OFF as 
desired. View additional detailed information about each analogy result. 


Data Viewer 


View detailed information for a single analogy result / technology project record. 
Display is optimized for human viewing, as opposed to Database sheet (see below) 
which is optimized for software performance. 


Data Importer 


Import Technology Project Data Collection Sheets into the TCASE database. 


Data Exporter 


Export data records from the TCASE database to the Technology Project Data Collection 
Sheet format. 


Database 


Collection of over 1,700 historical technology development project data records. 
Analogous projects are drawn from this database when user executes a search. 


Similarity Matrices 


User-defined matrices define relative similarity between different TRLs and between 
different System Flierarchy levels. Required for proper implementation of analogy 
matching algorithm. 
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Front End Worksheet 


The Front End worksheet serves as the basic interface with the TCASE tool. The sheet is neatly divided into 3 
sections: Search Criteria, Statistical Results, and Top Analogies. A screenshot can be found in Figure 2 on the 
following page. 

Search Criteria 

TCASE offers tremendous flexibility in the use of exact match filters and/or weighted search terms to enable you to 
obtain the most desirable set of analogy results. There are a few basic guidelines to keep in mind when inputting 
search criteria: 

• It is not necessary to fill out all of the Search Criteria search terms - you may use as many or as few as you 
like. As an extreme example, executing a search with no search terms will simply return the entire analogy 
database! 

• To enter multiple search terms in a single field, simply separate each with a space. Multiple terms will be 
processed as a Boolean OR search (e.g. a Description Keywords search for hydrogen rocket will identify 
projects that contain hydrogen and/or rocket in the description field]. 

• The Exact Match parameter can be set to either yes or no. If set to yes, then that particular search term will 
be treated as a filter, meaning that any project in the database that does not contain that search term in the 
specified field will not be returned as a viable result. 

• The Weighting parameter can be used to establish the relative importance of each search term (e.g. 
assigning a weighting = 1.0 for Title Keywords and a weighting of 2.0 for Primary TA would indicate that 
matching the Primary TA search term is twice as important as matching the Title Keywords). 

• The 2 parameters listed under Search Settings (Minimum Score for Analogy Matching and Maximum 
Number of Analogies in Results) are optional parameters that can be used to limit the number of results 
returned by a particular search. Lowering the minimum score or raising the maximum number of analogies 
will increase the number of results returned, and vice versa. 

Statistical Results 

The Statistical Results section will display summary statistics and graphical outputs based on the data returned by 
your most recent search. These results constitute some of the most significant outputs of the TCASE tool, so we’ll 
take a moment to review them in detail here: 

• Summary statistics can be found at the top of the Statistical Results section. These statistics are presented 
in 2 columns: 1 for cost and 1 for schedule. The count statistic indicates the number of analogies returned 
by your search that contain data for that metric (data for cost and/or data for schedule duration). Some 
records in the database may include cost data but not schedule data, or vice versa - thus, the 2 counts may 
be different numbers! Below the count statistic are some familiar measures of center and spread, including 
25 th /75 th percentile values, median value, mean value, maximum, and minimum. Finally, a box plot (aka 
"box-and-whiskers") is provided for each column. If you are unfamiliar with the box plot format, please 
refer to Figure 1 on the following page. 
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Figure 1. Definition of box plot as used in the TCASE tool. 

• Below the summary statistics, you will find several plots in the Analogy Breakdown section. These plots are 
intended to provide some contextual information about the analogy results returned by your search. At a 
glance you can see the prevalence of various Lead NASA Centers, System Hierarchy Levels, and Project Start 
Years within your results set. 

Top Analogies 

After a database search has been executed, the resulting analogous project results will be displayed in 2 locations 
in TCASE: (1) the Top Analogies section of the Front End, and (2] the Analogy Manager worksheet. The purpose of 
the Top Analogies sections is simply to give the user an overview, in list format, of the highest ranking analogous 
projects from the most recent search. Up to 50 of the highest ranking results can be displayed here. A 
comprehensive and more detailed list of the analogy results can be found on the Analogy Manager worksheet. 
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Figure 2. TCASE Front End Worksheet. 
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Analogy Manager Worksheet 


The Analogy Manager worksheet is a dynamic sheet containing a list of all analogous projects returned by the most 
recent user search. In some ways, the Analogy Manager is similar in function to a results webpage on Google or 
Bing. Whereas the Front End worksheet displays summary information such as statistics and up to 50 of the top 
analogy results, the Analogy Manager displays a simple list of all technology project analogies that met the user's 
search criteria. A screenshot of this worksheet is shown below in Figure 3. 



Figure 3. TCASE Analogy Manger Worksheet. 

In addition to containing a comprehensive list of results, the Analogy Manager worksheet includes a feature to 
include or exclude individual analogy results from the statistical calculations on the Front End sheet. In other 
words, if the user discovers one or more analogy results that he or she finds undesirable or inappropriate, these 
offending results may be omitted by selecting "N" (i.e. “No”) in the column labeled "Include”. The Summary 
Statistics section will instantly and automatically reflect changes due to any such data exclusions. 


Data Viewer Worksheet 


The Data Viewer worksheet offers a user friendly way to view the contents of a particular technology project data 
record from the database. The sheet contains a single input cell labeled "Enter Database ID”. Simply enter the 
Database ID corresponding to the project record in question, and all data from that record will be displayed in a 
clearly labeled and organized table. A screenshot of the Data Viewer is provided in Figure 4 on the following page. 

Note : The Data Viewer is a read-only utility and is not the appropriate place to attempt to edit a database record. 
The Database worksheet is the master location for all project data records. 
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Figure 4. TCASE Data Viewer Worksheet. 


Data Importer Worksheet 


From time to time it will be necessary to add new technology project data to the TCASE database. The Data 
Importer worksheet is designed to make this task much more efficient than manual data entry. The current version 
of TCASE is only capable of automatically importing data from specially formatted Technology Project Data 
Collection Sheets. Future versions of this application may be capable of interfacing with other input file formats. A 
screenshot of the Data Importer is provided below in Figure 5. 
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Figure 5. TCASE Data Importer Worksheet. 
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To use the Data Importer, start by clicking the 


Select Files to Import & r 


button. You should see a standard file 


browser interface. Navigate to the location of the Technology Project Data Collection Sheet or Sheets that you wish 
to import (you may select multiple files in the browser by holding the CTRL key while clicking) and then click 
Open. You will now see one or more rows of data appear in the Data Importer - 1 row corresponding with each 
imported project. 


TCASE will display a warning message if the name of any imported project appears to match the name of a project 
already in the database. Examine the column titled "Existing Entry in Database" to identify which project(s) may 
already exist. For each affected project, you may choose to either overwrite the existing data record in the database 
or create a new entry. 


Once you are satisfied with the project data to be imported, click the 
changes and/or additions to the TCASE database. 


Import Entries ► 


button to make the specified 


Data Exporter Worksheet 


The Data Exporter worksheet (see Figure 6) is designed to export one or more project data records from the 
TCASE database to a new Technology Project Data Collection Sheet. To begin using the Data Exporter, first open the 
Database worksheet in TCASE. To the left of each record in the database you will see a column entitled "Export 
Y/N". Simply identify the project record(s) you wish to export and change the value in the "Export" column to Y for 
each. 
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Figure 6. TCASE Data Exporter Worksheet. 


Once all desired projects have been identified and marked with a Y, open the Data Exporter worksheet and click 


Copy Selected Entries from Database fO 


the button. Doing so will copy each marked database record into the Data 

Exporter as a new row in the list of projects. Take a moment to review the "Export File Name” for each project. 
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These have been created automatically, but you are free to change the file name at your discretion. Once you are 


Export Entries ► 


satisfied with the list of projects to export and the export file names, simply click the button and 

a set of new Technology Project Data Collection Sheets will be created. Newly created Technology Project Data 
Collection Sheets will be saved in a subfolder within the directory where the TCASE Excel workbook resides. One 
sheet will be created for each technology project that you have exported. 


Database Worksheet 


As the name suggests, the Database worksheet is the core database that supports all other TCASE functionality. Out 
of the box, so to speak, the TCASE database contains more than 1,700 rows of data (aka "records" or "entries"] each 
of which includes up to 184 different data fields. The number of data records can be expanded - TCASE includes a 
Data Importer utility that allows a user to add new data records from Technology Project Data Collection Sheets. If 
you take a moment to browse the Database worksheet, you will notice that many data fields contain no data - 
perhaps half of all data fields in the database are empty. In some cases the empty cells indicate that that data field 
was not required for that particular project, while in other cases it represents incomplete information available 
about the project. As the TCASE research project continues, one objective is to improve the overall data quality in 
the database. A screenshot of a small portion of the TCASE database is shown below in Figure 7. 
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Figure 7. TCASE Database Worksheet. 


Practically speaking, a typical user will not have much need to access the Database worksheet directly. All analysis 
functions take place on the Front End and Analogy Manager worksheets. A more advanced user may wish to export 
data from the TCASE database using the Data Exporter described in the previous section. However, as we have 
seen even the Data Exporter requires only minimal user interaction with the database. 

It is possible to edit the TCASE database directly on the Database worksheet, although such edits must be 
conducted with great care and should only be attempted bv an experienced user with a backup file . For 

example, it is possible to delete an unwanted row from the TCASE database by clicking on the row number and 
using Excel's "Delete" function. It is also possible, although somewhat riskier, to insert a new row and manually fill 
it with valid data for a new project record. 
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Similarity Matrices Worksheet 


The Similarity Matrices worksheet contains 2 matrices used during the TCASE analogy matching process. These 
matrices are used to determine a degree of similarity between levels in the Technology Readiness Level (TRL) and 
System Hierarchy parameters. 

The default similarity matrix for TRL is show in Figure 8 below. Interpreting the matrix is fairly straightforward. If 
for example we were interested in determining the similarity score between a TRL of 4 and a TRL of 7, we would 
look for the intersection of row 4 and column 7 and find a value of 0.2. Values along the diagonal of the matrix are 
equal to 1.0 since they occur at the intersection of identical values. The similarity matrices are user-editable, if 
desired. 
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0.0 
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Figure 8. TCASE Similarity Matrix for TRL. 


Frequently Asked Questions 

• What is the origin of the technology development cost and schedule data in the TCASE database? 

The TCASE Database includes over 1,700 records from more than 60 sources. These sources include historical 
databases such as NTIS and MATCH, project records such as GCD and ETDP, SBIR records, and data solicited from 
technology managers during this current research period (2012-2013). More information can be found in 
Appendix B of this manual. 

• Has the cost data in the TCASE database been normalized? 

Yes. All costs in the TCASE database have been normalized to FY2013 dollars. 

• Does the TCASE tool utilize cost estimating relationships (CERs)? 

No, not at this time. TCASE uses an analogy-based approach to provide a likely range of cost and schedule for a 
particular type of technology development project. Future work plans call for a transition to CER-based 
parametric estimation methods if and when sufficient data is available to support that approach. 
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• My search returned more than 50 analogous projects, but only the top 50 are shown in the Top Analogies 
section. Where can I view the full results? 

Select the Analogy Manager worksheet to view the full list of analogous projects. 

• My search returned several projects that I believe are poor analogies for my estimating scenario. How can I 
improve the results? 

You have several options to refine your search. First, you might revise your search criteria, setting one or more 
of your search terms to "Exact Match" in order to impose a hard filter. Alternatively, you may view the Analogy 
Manager worksheet where you have the option to manually include or exclude each analogous project returned 
by TCASE. With a few clicks you can toss out one or more analogous projects that you feel are improperly 
skewing your results. The effects of any changes made on the Analogy Manager tab will be reflected in the 
results on the Front End as soon as you return to that worksheet. 
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Appendix A - Running a Simple Search 


1 . 

2 . 

3 . 

4 . 

5 . 


Open the MS Excel* workbook entitled TCASE Analogy Tool Beta v0.50.xlsm and access the 
Front End worksheet. 

Adjust parameters in the Search Criteria section as desired. Some parameters accept text 
inputs, others accept numerical inputs, and some are set via a pull-down menu. Relative 
weightings may be applied to your parameters to reflect their importance in your search. 

Click the button to run the TCASE search algorithm and generate analogous 

project results. * 

Review the outcome of your search in the Statistical Outputs section and the Top 
Analogies section of the Front End worksheet. 

To refine your search, simply edit the parameters in the Search Criteria section and click 
the button again. If you would like to start a completely new search, click the 

button and return to step 2 above. 


* Important Notes : The TCASE tool utilizes VBA macros during normal operation. Macros must be enabled in the 

user's local copy of Excel. Excel's Automatic Calculation setting must also be turned on. 
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Appendix B - Sources of Data Used in TCASE 

Early in the TCASE research project, the team identified more than 70 sources of historical technology project data. 
These sources included broad databases such as NTIS and MATCH, and also numerous individual project records 
from programs such as Game Changing Development (GCD) and the Exploration Technology Development 
Program (ETDP), to name a few. The table below lists some of the largest sources of technology project data. 


Short Name 

Description 

Year 

Number 
of Entries 

ETDP 

NASA Exploration Technology Development Program 

2012 

42 

ESTO 

NASA Earth Science Technology Office 

2012 

138 

SBIR 

Current NASA SBIR Technologies 

2012 

51 

GCD 

Game Changing Development Technology Projects 

2012 

35 

ESMD 

NASA Exploration Systems Mission Directorate ESAS 
Technologies 

2005 

304 

Tech Tool Box 

ATLAS (Advanced Technology Lifecycle Analysis System) 

2002 

6 

Ext Tech Data 

Tauri Group research into External Gov Technology Tech 
Maturation data 

2012 

28 

Cx ETDP 

Constellation Program Technology Portfolio 

2007 

19 

NTID 

NASA Tech Inventory Database 

2004 

991 

Hist SBIR STTR 

Historical SBIR and STTR data 

2012 

1191 

RLVTech DB 

NASA RLV Technology Database 

1993 

64 

HEOMD TechDev 
Team 

Estimates of HEOMD needed technology developments 

2012 

78 

ETDP from MATCH 

Mapping Applicable Technology To Exploration Challenges 

2007 

21 

External from MATCH 

Mapping Applicable Technology To Exploration Challenges 

2007 

192 

JSS TBCC 

NASA/Air Force Joint Systems Study TBCC Technologies 

2010 

18 


Not every technology project data record contained the critical cost and schedule duration values needed for use in 
TCASE. As indicated in the figure below, about half of the total records collected were suitable for use in TCASE (1,723 
out of 3,178). Thus, the database embedded in TCASE contains records for 1,723 historical and current technology 
development projects. 


3,178 

Historical Project 
Data Records 



1,723 

...of those include 

Cost and Schedule 

information 



643 * 
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